# Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

arXiv: https://arxiv.org/abs/2404.14901 · HTML: https://mlsanity.com/paper/2404.14901/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot](https://mlsanity.com/paper/2302.00438/) (2023)
- [Towards Human-Bot Collaborative Software Architecting with ChatGPT](https://mlsanity.com/paper/2302.14600/) (2023)
- [Better Together? An Evaluation of AI-Supported Code Translation](https://mlsanity.com/paper/2202.07682/) (2022)
- [A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions](https://mlsanity.com/paper/2305.00237/) (2023)
- [An Empirical Study of Bots in Software Development -- Characteristics and Challenges from a Practitioner's Perspective](https://mlsanity.com/paper/2005.13969/) (2020)
- [Automating Code-Related Tasks Through Transformers: The Impact of Pre-training](https://mlsanity.com/paper/2302.04048/) (2023)
- [Detecting Speech Act Types in Developer Question/Answer Conversations During Bug Repair](https://mlsanity.com/paper/1806.05130/) (2018)
- [Exploring Context-Aware Conversational Agents in Software Development](https://mlsanity.com/paper/2006.02370/) (2020)
- [AI-based Question Answering Assistance for Analyzing Natural-language Requirements](https://mlsanity.com/paper/2302.04793/) (2023)

## Built on by (most cited first, 25 of 25)

- [Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support](https://mlsanity.com/paper/2502.18658/) (2025)
- [AIDev: Studying AI Coding Agents on GitHub](https://mlsanity.com/paper/2602.09185/) (2026)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [LLMs: A Game-Changer for Software Engineers?](https://mlsanity.com/paper/2411.00932/) (2024)
- [mathbbUSCD: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding](https://mlsanity.com/paper/2409.05923/) (2024)
- [Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code](https://mlsanity.com/paper/2504.13656/) (2025)
- [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [WhatsCode: Large-Scale GenAI Deployment for Developer Efficiency at WhatsApp](https://mlsanity.com/paper/2512.05314/) (2025)
- [From Human-to-Human to Human-to-Bot Conversations in Software Engineering](https://mlsanity.com/paper/2405.12712/) (2024)
- [Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering](https://mlsanity.com/paper/2412.03815/) (2024)
- [HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding](https://mlsanity.com/paper/2505.22906/) (2025)
- [Beyond the Desk: Barriers and Future Opportunities for AI to Assist Scientists in Embodied Physical Tasks](https://mlsanity.com/paper/2603.19504/) (2026)
- [Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineering](https://mlsanity.com/paper/2604.10530/) (2026)
- [CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge](https://mlsanity.com/paper/2604.20389/) (2026)
- [An Empirical Study of Proactive Coding Assistants in Real-World Software Development](https://mlsanity.com/paper/2605.05700/) (2026)
- [Toward Neurosymbolic Program Comprehension](https://mlsanity.com/paper/2502.01806/) (2025)
- [Single Conversation Methodology: A Human-Centered Protocol for AI-Assisted Software Development](https://mlsanity.com/paper/2507.12665/) (2025)
- [Leveraging Large Language Models for Cybersecurity Risk Assessment -- A Case from Forestry Cyber-Physical Systems](https://mlsanity.com/paper/2510.06343/) (2025)
- [A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case](https://mlsanity.com/paper/2510.21933/) (2025)
- [Adoption of Generative Artificial Intelligence in the German Software Engineering Industry: An Empirical Study](https://mlsanity.com/paper/2601.16700/) (2026)
- [What Software Engineering Looks Like to AI Agents? -- An Empirical Study of AI-Only Technical Discourse on MoltBook](https://mlsanity.com/paper/2605.08380/) (2026)
- [AI-Generated Slides: Are They Good? Can Students Tell?](https://mlsanity.com/paper/2605.13532/) (2026)
- [Physically-Relevant Information Learning in High-Dimensional Time-Derivatives Spaces](https://mlsanity.com/paper/2607.05127/) (2026)
- [On the Prospects of Dynamic LLM Conversations in Software Development](https://mlsanity.com/paper/2608.30756/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user's personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.