mlsanityData to 14 Oct 2026

CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code

arXiv 2302.05527 ↗First posted 10 Feb 2023cs.SE cs.LG cs.PLcited by 102

Not on the map. Why?

Abstract (arXiv)

Since the rise of neural natural-language-to-code models (NL->Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has been reliably evaluating their generated output. In this paper, we propose CodeBERTScore: an evaluation metric for code generation, which builds on BERTScore (Zhang et al., 2020). Instead of encoding only the generated tokens as in BERTScore, CodeBERTScore also encodes the natural language input preceding the generated code, thus modeling the consistency between the generated code and its given natural language context as well. We perform an extensive evaluation of CodeBERTScore across four programming languages. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics. That is, generated code that receives a higher score by CodeBERTScore is more likely to be preferred by humans, as well as to function correctly when executed. We release five language-specific pretrained models to use with our publicly available code. Our language-specific models have been downloaded more than 1,000,000 times from the Huggingface Hub. Our code and data are available at https://github.com/neulab/code-bert-score

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

28 of 43 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
16 more
  1. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  2. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  3. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  4. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  5. 2018code2vec: Learning Distributed Representations of Codecited by 250
  6. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  7. 2018code2seq: Generating Sequences from Structured Representations of Codecited by 179
  8. 2023SantaCoder: don't reach for the stars!cited by 170
  9. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  10. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  11. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  12. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  13. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  14. 2021Self-Supervised Bug Detection and Repaircited by 59
  15. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 32
  16. 2022T5Score: Discriminative Fine-tuning of Generative Evaluation Metricscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 16; 2024: 29; 2025: 41; 2026: 16.

102 papers.

Sort:

By area:

  1. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  3. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  4. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  5. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  6. 2023Learning Performance-Improving Code Editscited by 106
  7. 2024CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counselingcited by 89
  8. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  9. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 57
  10. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  11. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 52
  12. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
38 more
  1. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 44
  2. 2024Human-In-the-Loop Software Development Agentscited by 42
  3. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  4. 2023Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChaincited by 31
  5. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  6. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  7. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 25
  8. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  9. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  10. 2025What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyondcited by 22
  11. 2024Consistent Autoformalization for Constructing Mathematical Librariescited by 21
  12. 2024SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Taskcited by 19
  13. 2024CodeRosetta: Pushing the Boundaries of Unsupervised Code Translation for Parallel Programmingcited by 19
  14. 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
  15. 2023Supersonic: Learning to Generate Source Code Optimizations in C/C++cited by 16
  16. 2024InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translationcited by 16
  17. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  18. 2023Exploring Distributional Shifts in Large Language Models for Code Analysiscited by 14
  19. 2023Context Aware Query Rewriting for Text Rankers using LLMcited by 14
  20. 2024What can Large Language Models Capture about Code Functional Equivalence?cited by 14
  21. 2025AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomycited by 13
  22. 2023QualEval: Qualitative Evaluation for Model Improvementcited by 11
  23. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 11
  24. 2024Model Editing for LLMs4Code: How Far are We?cited by 10
  25. 2025From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graphcited by 10
  26. 2024Output Format Biases in the Evaluation of Large Language Models for Code Translationcited by 9
  27. 2025Ensemble Learning for Large Language Models in Text and Code Generation: A Surveycited by 9
  28. 2024CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smellscited by 8
  29. 2025StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysiscited by 8
  30. 2025Towards Formal Verification of LLM-Generated Code from Natural Language Promptscited by 8
  31. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
  32. 2024Harnessing Business and Media Insights with Large Language Modelscited by 7
  33. 2024Efficient and Universal Watermarking for LLM-Generated Code Detectioncited by 6
  34. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  35. 2025Towards Better Understanding of Program-of-Thought Reasoning in Cross-Lingual and Multilingual Environmentscited by 6
  36. 2023AutoParLLM: GNN-guided Context Generation for Zero-Shot Code Parallelization using LLMscited by 5
  37. 2024A Transformer-Based Approach for Smart Invocation of Automatic Code Completioncited by 5
  38. 2025Improving LoRA with Variational Learningcited by 5

50 most cited of 102 All 102 (JSON)