mlsanityData to 7 Oct 2026

Out of the BLEU: how should we assess quality of the Code Generation models?

arXiv 2208.03133 ↗First posted 5 Aug 2022cs.SE cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

In recent years, researchers have created and introduced a significant number of various code generation models. As human evaluation of every new model version is unfeasible, the community adopted automatic evaluation metrics such as BLEU to approximate the results of human judgement. These metrics originate from the machine translation domain and it is unclear whether they are applicable for the code generation tasks and how well they agree with the human evaluation on this task. There are also other metrics, CodeBLEU and RUBY, developed to estimate the similarity of code, that take into account the properties of source code. However, for these metrics there are hardly any studies on their agreement with the human evaluation. Despite all that, minimal differences in the metric scores have been used in recent papers to claim superiority of some code generation models over the others. In this paper, we present a study on the applicability of six metrics -- BLEU, ROUGE-L, METEOR, ChrF, CodeBLEU, and RUBY -- for evaluation of code generation models. We conduct a study on two different code generation datasets and use human annotators to assess the quality of all models run on these datasets. The results indicate that for the CoNaLa dataset of Python one-liners, none of the metrics can correctly emulate human judgement on which model is better with >95% certainty if the difference in model scores is less than 5 points. For the HearthStone dataset, which consists of classes of a particular structure, a difference in model scores of at least 2 points is enough to claim the superiority of one model over the other. Our findings suggest that the ChrF metric is a better fit for the evaluation of code generation models than the commonly used BLEU and CodeBLEU. Yet, finding a metric for code generation that closely agrees with humans requires additional work.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 22 of its 53 references that are in the map. First posted on arXiv 5 Aug 2022.

Builds on

22 of 53 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  8. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  9. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  10. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  11. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  12. 2018Mapping Language to Code in Programmatic Contextcited by 104
10 more
  1. 2017Abstract Syntax Networks for Code Generation and Semantic Parsingcited by 92
  2. 2018TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generationcited by 88
  3. 2016Latent Predictor Networks for Code Generationcited by 73
  4. 2019Code Generation as a Dual Task of Code Summarizationcited by 62
  5. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 56
  6. 2019Does BLEU Score Work for Code Migration?cited by 49
  7. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  8. 2019Recommendations for Datasets for Source Code Summarizationcited by 46
  9. 2018A Grammar-Based Structural CNN Decoder for Code Generationcited by 29
  10. 2020Adversarial Crowdsourcing Through Robust Rank-One Matrix Completioncited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 15; 2025: 26; 2026: 8.

65 papers.

Sort:

By area:

  1. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  2. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  3. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  4. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  5. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  6. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  7. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 41
  8. 2024Language Models for Code Completion: A Practical Evaluationcited by 38
  9. 2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
  10. 2024Consistent Autoformalization for Constructing Mathematical Librariescited by 20
  11. 2024NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Promptscited by 19
  12. 2024SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Taskcited by 19
38 more
  1. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 19
  2. 2024CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programmingcited by 19
  3. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  4. 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
  5. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  6. 2023Exploring Distributional Shifts in Large Language Models for Code Analysiscited by 14
  7. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 12
  8. 2022Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generatorscited by 10
  9. 2023A Deep Learning Framework for Verilog Autocompletion Towards Design and Verification Automationcited by 10
  10. 2024When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inferencecited by 10
  11. 2024CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smellscited by 8
  12. 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
  13. 2023FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repaircited by 6
  14. 2023Automating the Correctness Assessment of AI-generated Code for Security Contextscited by 6
  15. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  16. 2025CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generationcited by 6
  17. 2025CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMscited by 6
  18. 2022SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Modelscited by 5
  19. 2024A Transformer-Based Approach for Smart Invocation of Automatic Code Completioncited by 5
  20. 2025SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solvingcited by 4
  21. 2025Code Roulette: How Prompt Variability Affects LLM Code Generationcited by 4
  22. 2025Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judgecited by 4
  23. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarkingcited by 4
  24. 2024OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agentscited by 3
  25. 2025GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generationcited by 3
  26. 2025Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insightscited by 3
  27. 2025ChronoLLM: Customizing Language Models for Physics-Based Simulation Code Generationcited by 3
  28. 2024Test Code Generation for Telecom Software Systems using Two-Stage Generative Modelcited by 2
  29. 2024DocuMint: Docstring Generation for Python using Small Language Modelscited by 2
  30. 2025An Extensive Evaluation of PDDL Capabilities in off-the-shelf LLMscited by 2
  31. 2025DDPT: Diffusion-Driven Prompt Tuning for Large Language Model Code Generationcited by 2
  32. 2025Measuring LLM Code Generation Stability via Structural Entropycited by 2
  33. 2025Large Language Models for Software Testing: A Research Roadmapcited by 2
  34. 2025Mellum: Production-Grade in-IDE Contextual Code Completion with Multi-File Project Understandingcited by 2
  35. 2023Dynamic Retrieval-Augmented Generationcited by 1
  36. 2024Retrieval-augmented code completion for local projects using large language modelscited by 1
  37. 2025Style2Code: A Style-Controllable Code Generation Framework with Dual-Modal Contrastive Representation Learningcited by 1
  38. 2025LaajMeter: A Framework for LaaJ Evaluationcited by 1

50 most cited of 65 All 65 (JSON)