mlsanityData to 7 Oct 2026

CodeScore: Evaluating Code Generation by Learning Code Execution

arXiv 2301.09043 ↗First posted 22 Jan 2023cs.SEcited by 66

Not on the map. Why?

Abstract (arXiv)

A proper code evaluation metric (CEM) profoundly impacts the evolution of code generation, which is an important research field in NLP and software engineering. Prevailing match-based CEMs (e.g., BLEU, Accuracy, and CodeBLEU) suffer from two significant drawbacks. 1. They primarily measure the surface differences between codes without considering their functional equivalence. However, functional equivalence is pivotal in evaluating the effectiveness of code generation, as different codes can perform identical operations. 2. They are predominantly designed for the Ref-only input format. However, code evaluation necessitates versatility in input formats. Aside from Ref-only, there are NL-only and Ref&NL formats, which existing match-based CEMs cannot effectively accommodate. In this paper, we propose CodeScore, a large language model (LLM)-based CEM, which estimates the functional correctness of generated code on three input types. To acquire CodeScore, we present UniCE, a unified code generation learning framework, for LLMs to learn code execution (i.e., learning PassRatio and Executability of generated code) with unified input. Extensive experimental results on multiple code evaluation datasets demonstrate that CodeScore absolutely improves up to 58.87% correlation with functional correctness compared to other CEMs, achieves state-of-the-art performance, and effectively handles three input formats.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jan 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 21; 2025: 23; 2026: 15.

66 papers.

Sort:

By area:

  1. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  2. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  3. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  4. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  5. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  6. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  7. 2024SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agentscited by 34
  8. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  9. 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisationcited by 31
  10. 2025CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debuggingcited by 31
  11. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  12. 2024RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generationcited by 25
38 more
  1. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 22
  2. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
  3. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 15
  4. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  5. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  6. 2024What can Large Language Models Capture about Code Functional Equivalence?cited by 13
  7. 2025Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Modelcited by 13
  8. 2024RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidancecited by 12
  9. 2024Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Modelscited by 10
  10. 2025Execution Guided Line-by-Line Code Generationcited by 10
  11. 2024OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Modelscited by 8
  12. 2024Harnessing Business and Media Insights with Large Language Modelscited by 7
  13. 2024Planning-Driven Programming: A Large Language Model Programming Workflowcited by 7
  14. 2025Large Language Model Guided Self-Debugging Code Generationcited by 7
  15. 2025Efficient LLM Collaboration via Planningcited by 7
  16. 2023FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repaircited by 6
  17. 2023Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generationcited by 6
  18. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  19. 2025RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generationcited by 5
  20. 2026SEMAG: Self-Evolutionary Multi-Agent Code Generationcited by 5
  21. 2024DOCE: Finding the Sweet Spot for Execution-Based Code Generationcited by 4
  22. 2024CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validationcited by 4
  23. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  24. 2025NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debuggingcited by 4
  25. 2025GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Modelscited by 4
  26. 2025MERA Code: A Unified Framework for Evaluating Code Generation Across Taskscited by 4
  27. 2025When Names Disappear: Revealing What LLMs Actually Understand About Codecited by 4
  28. 2026Multi-agent Collaboration with State Managementcited by 4
  29. 2023Neural Machine Translation for Code Generationcited by 3
  30. 2024Exploring Data-Efficient Adaptation of Large Language Models for Code Generationcited by 3
  31. 2024Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinementcited by 3
  32. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  33. 2025Understanding and Bridging the Planner-Coder Gap: A Systematic Study on the Robustness of Multi-Agent Systems for Code Generationcited by 3
  34. 2026CodeCircuit: Toward Inferring LLM-Generated Code Correctness via Attribution Graphscited by 3
  35. 2024Exploring Language Model's Code Generation Ability with Auxiliary Functionscited by 2
  36. 2025MoT: Modularization-of-Thought Prompting for Effective Code Generationcited by 2
  37. 2025Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repaircited by 2
  38. 2026RefineEvo: Planning-Guided Heuristic Evolution with Bidirectional Experiencecited by 2

50 most cited of 66 All 66 (JSON)