mlsanityData to 7 Oct 2026

Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues

arXiv 2307.12596 ↗First posted 24 Jul 2023cs.SEcited by 78

Not on the map. Why?

Abstract (arXiv)

We systematically study the quality of 4,066 ChatGPT-generated code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is three folds. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT's self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of AI models like ChatGPT.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 20; 2025: 33; 2026: 20.

78 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2024A Survey on Large Language Models for Code Generationcited by 646
  3. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  4. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  5. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  6. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  7. 2024Human-In-the-Loop Software Development Agentscited by 42
  8. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  9. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 34
  10. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  11. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  12. 2023Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmapscited by 19
38 more
  1. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
  2. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 16
  3. 2025Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inferencecited by 14
  4. 2025Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarioscited by 14
  5. 2025Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimizationcited by 14
  6. 2025SandboxEval: Towards Securing Test Environment for Untrusted Codecited by 10
  7. 2025Security Degradation in Iterative AI Code Generation -- A Systematic Analysis of the Paradoxcited by 10
  8. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9
  9. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
  10. 2025Investigating The Smells of LLM Generated Codecited by 9
  11. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  12. 2024A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llamacited by 7
  13. 2024Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AIcited by 6
  14. 2025Exploring Large Language Models for Semantic Analysis and Categorization of Android Malwarecited by 6
  15. 2025Multi-Agent Collaboration for Multilingual Code Instruction Tuningcited by 6
  16. 2024Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testingcited by 5
  17. 2024Evaluating AI-generated code for C++, Fortran, Go, Java, Julia, Matlab, Python, R, and Rustcited by 5
  18. 2024Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmerscited by 5
  19. 2024AIGCodeSet: A New Annotated Dataset for AI Generated Code Detectioncited by 5
  20. 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-benchcited by 5
  21. 2025Turning the Tide: Repository-based Code Reflectioncited by 5
  22. 2023LUNA: A Model-Based Universal Analysis Framework for Large Language Modelscited by 4
  23. 2025Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassiancited by 4
  24. 2025A Benchmark for Localizing Code and Non-Code Issues in Software Projectscited by 4
  25. 2024Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMscited by 3
  26. 2025Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategiescited by 3
  27. 2026Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agentscited by 3
  28. 2025MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarioscited by 2
  29. 2025Hallucination in LLM-Based Code Generation: An Automotive Case Studycited by 2
  30. 2025IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injectioncited by 2
  31. 2026An Exploratory Study on LLM-Generated Code and Comments in Code Repositoriescited by 2
  32. 2024Morescient GAI for Software Engineering (Extended Version)cited by 1
  33. 2024The Rise and Fall(?) of Software Engineeringcited by 1
  34. 2024Protect Your Secrets: Understanding and Measuring Data Exposure in VSCode Extensionscited by 1
  35. 2025Autonomous Legacy Web Application Upgrades Using a Multi-Agent Systemcited by 1
  36. 2025Enhancing the Non-Functional Quality Compliance of LLM-Generated Code through Quality-Aware Preference Learningcited by 1
  37. 2025On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practicescited by 1
  38. 2025From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMscited by 1

50 most cited of 78 All 78 (JSON)