mlsanityData to 7 Oct 2026

Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review

arXiv 2406.12655 ↗First posted 18 Jun 2024cs.AI cs.SEcited by 19

Not on the map. Why?

Abstract (arXiv)

With the rapid development of Large Language Models (LLMs), a large number of machine learning models have been developed to assist programming tasks including the generation of program code from natural language input. However, how to evaluate such LLMs for this task is still an open problem despite of the great amount of research efforts that have been made and reported to evaluate and compare them. This paper provides a critical review of the existing work on the testing and evaluation of these tools with a focus on two key aspects: the benchmarks and the metrics used in the evaluations. Based on the review, further research directions are discussed.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

26 of 47 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2023StarCoder: may the source be with you!cited by 790
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2019SPoC: Search-based Pseudocode to Codecited by 176
  12. 2023SantaCoder: don't reach for the stars!cited by 169
14 more
  1. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  2. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  3. 2018Mapping Language to Code in Programmatic Contextcited by 104
  4. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  5. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  6. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  7. 2019Does BLEU Score Work for Code Migration?cited by 49
  8. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  9. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 41
  10. 2023Distilled GPT for Source Code Summarizationcited by 30
  11. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23
  12. 2023Deep Learning for Code Intelligence: Survey, Benchmark and Toolkitcited by 22
  13. 2024Automated Program Repair: Emerging trends pose and expose problems for benchmarkscited by 7
  14. 2023A Comprehensive Review of State-of-The-Art Methods for Java Code Generation from Natural Language Textcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  2. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  3. 2025Investigating The Smells of LLM Generated Codecited by 9
  4. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
  5. 2026MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistantscited by 7
  6. 2024Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmerscited by 5
  7. 2025The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMscited by 5
  8. 2025SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completioncited by 3
  9. 2025Measuring LLM Code Generation Stability via Structural Entropycited by 2
  10. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 2
  11. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  12. 2025CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluationcited by 1
7 more
  1. 2025MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Taskscited by 1
  2. 2025GenDexHand: Generative Simulation for Dexterous Handscited by 1
  3. 2025Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function and Project-level Code Generation Taskscited by 0
  4. 2025app.build: A Production Framework for Scaling Agentic Prompt-to-App Generation with Environment Scaffoldingcited by 0
  5. 2025Designing and Evaluating Chain-of-Hints for Scientific Question Answeringcited by 0
  6. 2026Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Studycited by 0
  7. 2026SkillComposer: Learning Reusable Skills for Natural-Language Robot Programmingcited by 0