mlsanityData to 7 Oct 2026

On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation

arXiv 2405.01580 ↗First posted 26 Apr 2024cs.SE cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

The task of code generation from natural language (NL2Code) has become extremely popular, especially with the advent of Large Language Models (LLMs). However, efforts to quantify and track this progress have suffered due to a lack of reliable metrics for functional correctness. While popular benchmarks like HumanEval have test cases to enable reliable evaluation of correctness, it is time-consuming and requires human effort to collect test cases. As an alternative several reference-based evaluation metrics have been proposed, with embedding-based metrics like CodeBERTScore being touted as having a high correlation with human preferences and functional correctness. In our work, we analyze the ability of embedding-based metrics like CodeBERTScore to measure functional correctness and other helpful constructs like editing effort by analyzing outputs of ten models over two popular code generation benchmarks. Our results show that while they have a weak correlation with functional correctness (0.16), they are strongly correlated (0.72) with editing effort.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 57 of its 78 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

57 of 78 references.

References · 57

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
38 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  3. 2021Measuring Coding Challenge Competence With APPScited by 879
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2023StarCoder: may the source be with you!cited by 790
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  7. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  8. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  10. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  11. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  12. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  13. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  14. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  15. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  16. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  17. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  18. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  19. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  20. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  21. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  22. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  23. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  24. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 123
  25. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  26. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 89
  27. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  28. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  29. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 67
  30. 2023CodeScore: Evaluating Code Generation by Learning Code Executioncited by 66
  31. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  32. 2021SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representationcited by 63
  33. 2018Neural Code Comprehension: A Learnable Representation of Code Semanticscited by 59
  34. 2019Does BLEU Score Work for Code Migration?cited by 49
  35. 2019JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generationcited by 49
  36. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  37. 2019Program Synthesis and Semantic Parsing with Learned Code Idiomscited by 34
  38. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruningcited by 19
  2. 2025LLM-Supported Natural Language to Bash Translationcited by 2
  3. 2025CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluationcited by 1
  4. 2025ReCatcher: Towards LLMs Regression Testing for Code Generationcited by 0
  5. 2025Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilitiescited by 0
  6. 2026Noisy Test-Time Reinforcement Learning for Code LLMscited by 0