# Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation

arXiv: https://arxiv.org/abs/2402.11702 · HTML: https://mlsanity.com/paper/2402.11702/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 9 of 9)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022)
- [ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks](https://mlsanity.com/paper/2305.16837/) (2023)

## Built on by (most cited first, 11 of 11)

- [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025)
- [How Do Programming Students Use Generative AI?](https://mlsanity.com/paper/2501.10091/) (2025)
- [Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations](https://mlsanity.com/paper/2408.05128/) (2024)
- [TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code](https://mlsanity.com/paper/2602.06875/) (2026)
- [Automated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithm](https://mlsanity.com/paper/2408.11198/) (2024)
- [Intuition to Evidence: Measuring AI's True Impact on Developer Productivity](https://mlsanity.com/paper/2509.19708/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow](https://mlsanity.com/paper/2507.10818/) (2025)
- [On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practices](https://mlsanity.com/paper/2504.16485/) (2025)
- [LLM Assisted Coding with Metamorphic Specification Mutation Agent](https://mlsanity.com/paper/2511.18249/) (2025)
- [An Exploratory Case Study of LLM-Assisted Refactoring and Gameplay Feature Generation in an Endless Runner Game](https://mlsanity.com/paper/2606.21171/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated notable proficiency in code generation, with numerous prior studies showing their promising capabilities in various development scenarios. However, these studies mainly provide evaluations in research settings, which leaves a significant gap in understanding how effectively LLMs can support developers in real-world. To address this, we conducted an empirical analysis of conversations in DevGPT, a dataset collected from developers' conversations with ChatGPT (captured with the Share Link feature on platforms such as GitHub). Our empirical findings indicate that the current practice of using LLM-generated code is typically limited to either demonstrating high-level concepts or providing examples in documentation, rather than to be used as production-ready code. These findings indicate that there is much future work needed to improve LLMs in code generation before they can be integral parts of modern software development.