mlsanityData to 7 Oct 2026

Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT

arXiv 2304.10778 ↗First posted 21 Apr 2023cs.SEcited by 71

Not on the map. Why?

Abstract (arXiv)

Context: AI-assisted code generation tools have become increasingly prevalent in software engineering, offering the ability to generate code from natural language prompts or partial code inputs. Notable examples of these tools include GitHub Copilot, Amazon CodeWhisperer, and OpenAI's ChatGPT. Objective: This study aims to compare the performance of these prominent code generation tools in terms of code quality metrics, such as Code Validity, Code Correctness, Code Security, Code Reliability, and Code Maintainability, to identify their strengths and shortcomings. Method: We assess the code generation capabilities of GitHub Copilot, Amazon CodeWhisperer, and ChatGPT using the benchmark HumanEval Dataset. The generated code is then evaluated based on the proposed code quality metrics. Results: Our analysis reveals that the latest versions of ChatGPT, GitHub Copilot, and Amazon CodeWhisperer generate correct code 65.2%, 46.3%, and 31.1% of the time, respectively. In comparison, the newer versions of GitHub CoPilot and Amazon CodeWhisperer showed improvement rates of 18% for GitHub Copilot and 7% for Amazon CodeWhisperer. The average technical debt, considering code smells, was found to be 8.9 minutes for ChatGPT, 9.1 minutes for GitHub Copilot, and 5.6 minutes for Amazon CodeWhisperer. Conclusions: This study highlights the strengths and weaknesses of some of the most popular code generation tools, providing valuable insights for practitioners. By comparing these generators, our results may assist practitioners in selecting the optimal tool for specific tasks, enhancing their decision-making process.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Apr 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 21; 2025: 21; 2026: 21.

71 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  3. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  4. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  6. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  7. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  8. 2024Lean Copilot: Large Language Models as Copilots for Theorem Proving in Leancited by 52
  9. 2023SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Supportcited by 46
  10. 2024SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agentscited by 34
  11. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  12. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 33
38 more
  1. 2024Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmerscited by 27
  2. 2024The Future of AI-Driven Software Engineeringcited by 23
  3. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  4. 2024Large Action Models: From Inception to Implementationcited by 18
  5. 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 18
  6. 2025AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvementcited by 13
  7. 2024RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidancecited by 12
  8. 2024Enhancing Large Language Models with Pseudo- and Multisource- Knowledge Graphs for Open-ended Question Answeringcited by 8
  9. 2024Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generationcited by 7
  10. 2024Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Studycited by 7
  11. 2025Evaluating Large Language Models for Code Reviewcited by 7
  12. 2025Pragmatic Reasoning improves LLM Code Generationcited by 6
  13. 2025LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languagescited by 5
  14. 2026HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Executioncited by 5
  15. 2023An Empirical Study of AI-based Smart Contract Creationcited by 4
  16. 2023Language Models for Business Optimisation with a Real World Case Study in Production Schedulingcited by 4
  17. 2025AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Softwarecited by 4
  18. 2026Revisiting DAgger in the Era of LLM-Agentscited by 4
  19. 2024TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Modelscited by 3
  20. 2024Controlled Automatic Task-Specific Synthetic Data Generation for Hallucination Detectioncited by 3
  21. 2024Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarkscited by 3
  22. 2025GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generationcited by 3
  23. 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Timecited by 3
  24. 2026Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agentscited by 3
  25. 2024The Impact of AI Tool on Engineering at ANZ Bank An Empirical Study on GitHub Copilot within Corporate Environmentcited by 2
  26. 2024A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Modelscited by 2
  27. 2025Unlocking the Potential of Past Research: Using Generative AI to Reconstruct Healthcare Simulation Modelscited by 2
  28. 2025Adversarial Reasoning for Repair Based on Inferred Program Intentcited by 2
  29. 2025Do Generative AI Tools Ensure Green Code? An Investigative Studycited by 2
  30. 2025Pre-Filtering Code Suggestions using Developer Behavioral Telemetry to Optimize LLM-Assisted Programmingcited by 2
  31. 2026The State of Generative AI in Software Development: Insights from Literature and a Developer Surveycited by 2
  32. 2025The Role of GitHub Copilot on Software Development: A Perspective on Productivity, Security, Best Practices and Future Directionscited by 1
  33. 2025Enhancing the Non-Functional Quality Compliance of LLM-Generated Code through Quality-Aware Preference Learningcited by 1
  34. 2025Exploring utilization of generative AI for research and education in data-driven materials sciencecited by 1
  35. 2025On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practicescited by 1
  36. 2025Investigating Security Implications of Automatically Generated Code on the Software Supply Chaincited by 1
  37. 2025Secure Code Generation at Scale with Reflexioncited by 1
  38. 2026Social Bias in LLM-Generated Code: Benchmark and Mitigationcited by 1

50 most cited of 71 All 71 (JSON)