mlsanityData to 7 Oct 2026

No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT

arXiv 2308.04838 ↗First posted 9 Aug 2023cs.SEcited by 45

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities across various NLP tasks. Additionally, LLMs are also highly valuable in supporting software engineering tasks, particularly in the field of code generation. Automatic code generation is a process of automatically generating source code or executable code based on given specifications or requirements, improving developer productivity. In this study, we perform a systematic empirical assessment to the quality of code generation using ChatGPT. We leverage 728 algorithm problems in five languages (i.e., C, C++, Java, Python, and JavaScript) and 18 CWEs with 54 code scenarios for the code generation task. Our evaluation encompasses a comprehensive analysis of code snippets generated by ChatGPT, focusing on three critical aspects: correctness, complexity, and security. We also specifically investigate ChatGPT's ability to engage in multi-round fixing process (i.e., ChatGPT's dialog ability) of facilitating code generation. By delving into the generated code and examining the experimental results, this work provides valuable insights into the performance of ChatGPT in tackling code generation tasks over the three critical aspects. Overall, our findings uncover potential issues and limitations that arise in the ChatGPT-based code generation and lay the groundwork for improving AI and LLM-based code generation techniques.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Aug 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 17; 2026: 11.

45 papers.

Sort:

By area:

  1. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  2. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  3. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  4. 2025On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHubcited by 53
  5. 2024An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detectioncited by 32
  6. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  7. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  8. 2024On the Impacts of Contexts on Repository-Level Code Generationcited by 21
  9. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  10. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 16
  11. 2025Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarioscited by 14
  12. 2024Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answerscited by 12
33 more
  1. 2024LLMs: A Game-Changer for Software Engineers?cited by 11
  2. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  3. 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 9
  4. 2025Generative AI and Empirical Software Engineering: A Paradigm Shiftcited by 9
  5. 2025Large Language Models for Security Operations Centers: A Comprehensive Surveycited by 9
  6. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  7. 2024How Well Do Large Language Models Serve as End-to-End Secure Code Agents for Python?cited by 7
  8. 2025DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generationcited by 7
  9. 2025Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspectivecited by 6
  10. 2023Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Codecited by 4
  11. 2023LLM4TDD: Best Practices for Test Driven Development Using Large Language Modelscited by 4
  12. 2025Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projectscited by 4
  13. 2025SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solvingcited by 4
  14. 2024Questionable practices in machine learningcited by 3
  15. 2024One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Taskscited by 3
  16. 2024A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicalitycited by 2
  17. 2025Secure-Instruct: An Automated Pipeline for Synthesizing Instruction-Tuning Datasets Using LLMs for Secure Code Generationcited by 2
  18. 2025On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practicescited by 1
  19. 2025SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysiscited by 1
  20. 2025Investigating Security Implications of Automatically Generated Code on the Software Supply Chaincited by 1
  21. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  22. 2026FASE: Fast Adaptive Semantic Entropy for Code Qualitycited by 1
  23. 2024The why, what, and how of AI-based coding in scientific researchcited by 0
  24. 2025Uncovering Code Insights: Leveraging GitHub Artifacts for Deeper Code Understandingcited by 0
  25. 2026STELP: Secure Transpilation and Execution of LLM-Generated Programscited by 0
  26. 2026Security-by-Design for LLM-Based Code Generation: Leveraging Internal Representations for Concept-Driven Steering Mechanismscited by 0
  27. 2026An Exploratory Case Study of LLM-Assisted Refactoring and Gameplay Feature Generation in an Endless Runner Gamecited by 0
  28. 2026Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystemcited by 0
  29. 2026Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Studycited by 0
  30. 2026Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Studycited by 0
  31. 2026Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphscited by 0
  32. 2026SynH-Rank: Quality-Aware Code Search via Diverse Data Synthesis and Hierarchical Ranking Trainingcited by 0
  33. 2026Adaptive Critical Token-Aware Retrieval for Repository-Level Code Generationcited by 0