mlsanityData to 7 Oct 2026

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

arXiv 2305.01210 ↗First posted 2 May 2023cs.SE cs.CL cs.LGcited by 1,332

Not on the map. Why?

Cited by 182 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.

Key facts. Cited by 1,332 papers in the map (data through 7 Oct 2026). Builds on 31 of its 93 references that are in the map. First posted on arXiv 2 May 2023.

Builds on

31 of 93 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  9. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  12. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
19 more
  1. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  2. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  3. 2020Unsupervised Translation of Programming Languagescited by 262
  4. 2023SantaCoder: don't reach for the stars!cited by 169
  5. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  6. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  7. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  8. 2018Mapping Language to Code in Programmatic Contextcited by 104
  9. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 102
  10. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  11. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  12. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  13. 2018Neural-Guided Deductive Search for Real-Time Program Synthesis from Examplescited by 39
  14. 2022Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testingcited by 36
  15. 2023Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTcited by 36
  16. 2023Learning Deep Semantics for Test Completioncited by 31
  17. 2021Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzingcited by 12
  18. 2022Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutationcited by 9
  19. 2021Learning to Superoptimize Real-world Programscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 69; 2024: 305; 2025: 428; 2026: 530.

1,332 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2024Phi-4 Technical Reportcited by 675
  12. 2024A Survey on Large Language Models for Code Generationcited by 646
38 more
  1. 2023Textbooks Are All You Needcited by 561
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  3. 2025Kimi K2: Open Agentic Intelligencecited by 523
  4. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  5. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  7. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  8. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  9. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  10. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  11. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  12. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  13. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  14. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  15. 2023Using an LLM to Help With Code Understandingcited by 222
  16. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  17. 2025Learning to Reason without External Rewardscited by 199
  18. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  19. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  20. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  21. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  22. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  23. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  24. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  25. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  26. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  27. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  28. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  29. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  30. 2026MiMo-V2-Flash Technical Reportcited by 163
  31. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  32. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  33. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  34. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  35. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  36. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  37. 2025Phi-4-reasoning Technical Reportcited by 146
  38. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143

50 most cited of 1,332 All 1,332 (JSON)