mlsanityData to 14 Oct 2026

Aligning Offline Metrics and Human Judgments of Value for Code Generation Models

arXiv 2210.16494 ↗First posted 29 Oct 2022cs.SE cs.AI cs.HCcited by 14

Not on the map. Why?

Abstract (arXiv)

Large language models have demonstrated great potential to assist programmers in generating code. For such human-AI pair programming scenarios, we empirically demonstrate that while generated code is most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide. Through a user study with N = 49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task. Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

26 of 39 references.

References · 26

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  6. 2021Measuring Coding Challenge Competence With APPScited by 902
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 366
  12. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
14 more
  1. 2019Measurement and Fairnesscited by 308
  2. 2020Unsupervised Translation of Programming Languagescited by 263
  3. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  4. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  5. 2019SPoC: Search-based Pseudocode to Codecited by 176
  6. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  7. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 126
  8. 2022Productivity Assessment of Neural Code Completioncited by 98
  9. 2022The Fallacy of AI Functionalitycited by 88
  10. 2017A parallel corpus of Python functions and documentation strings for automated code documentation and code generationcited by 57
  11. 2021Perfection Not Required? Human-AI Partnerships in Code Translationcited by 42
  12. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  13. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  14. 2014Learning Natural Coding Conventionscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 4; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  2. 2024CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledgecited by 55
  3. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 30
  4. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  5. 2024CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarkscited by 23
  6. 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 16
  7. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  8. 2024Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applicationscited by 5
  9. 2025The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMscited by 5
  10. 2026ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenascited by 3
  11. 2025David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generationcited by 1
  12. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
2 more
  1. 2025Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilitiescited by 0
  2. 2026Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
  2. 2026More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests
  3. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representations
  4. 2025Human-Aligned Code Readability Assessment with Large Language Models ↗
  5. 2024Evaluating and Aligning CodeLLMs on Human Preference
  6. 2026Code Is More Than Text: Uncertainty Estimation for Code Generation
  7. 2025SWE-IF: Aligning Code Evaluation with Human Preference
  8. 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling