mlsanityData to 7 Oct 2026

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

arXiv 2009.10297 ↗First posted 22 Sep 2020cs.SE cs.CLcited by 468

Not on the map. Why?

Cited by 44 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models. In the area of code synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but they are not suitable enough to evaluate codes, because BLEU is originally designed to evaluate the natural language, neglecting important syntactic and semantic features of codes, and perfect accuracy is too strict thus it underestimates different outputs with the same semantic logic. To remedy this, we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the strength of BLEU in the n-gram match and further injects code syntax via abstract syntax trees (AST) and code semantics via data-flow. We conduct experiments by evaluating the correlation coefficient between CodeBLEU and quality scores assigned by the programmers on three code synthesis tasks, i.e., text-to-code, code translation, and code refinement. Experimental results show that our proposed CodeBLEU can achieve a better correlation with programmer assigned scores compared with BLEU and accuracy.

Key facts. Cited by 468 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 38; 2023: 70; 2024: 107; 2025: 145; 2026: 95.

468 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  3. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2024A Survey on Large Language Models for Code Generationcited by 646
  7. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  8. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  9. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  10. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  12. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 281
38 more
  1. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  2. 2023SantaCoder: don't reach for the stars!cited by 169
  3. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  4. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  5. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  6. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  7. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  8. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  9. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  10. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  11. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 93
  12. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  13. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  14. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  15. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  16. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
  17. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  18. 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
  19. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  20. 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvementcited by 57
  21. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
  22. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  23. 2024Repoformer: Selective Retrieval for Repository-Level Code Completioncited by 55
  24. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 54
  25. 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 53
  26. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  27. 2024Multi-objective Evolution of Heuristic Using Large Language Modelcited by 52
  28. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  29. 2022CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Modelscited by 48
  30. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  31. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 46
  32. 2024WebCode2M: A Real-World Dataset for Code Generation from Webpage Designscited by 43
  33. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  34. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  35. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  36. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 41
  37. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  38. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39

50 most cited of 468 All 468 (JSON)