mlsanityData to 7 Oct 2026

FixEval: Execution-based Evaluation of Program Fixes for Programming Problems

arXiv 2206.07796 ↗First posted 15 Jun 2022cs.SE cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs. In response, researchers have explored various methods to automatically generate fixes for buggy code. However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively. To address this issue, we introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes and assess further information regarding time, memory constraints, and acceptance based on a verdict. We consider two Transformer language models pretrained on programming languages as our baseline and compare them using match-based and execution-based evaluation metrics. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately. At the same time, execution-based methods evaluate programs through all cases and scenarios designed explicitly for that solution. Therefore, we believe FixEval provides a step towards real-world automatic bug fixing and model-generated code evaluation. The dataset and models are open-sourced at https://github.com/mahimanzum/FixEval.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 20 of its 48 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

20 of 48 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  8. 2021Measuring Coding Challenge Competence With APPScited by 879
  9. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  12. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
8 more
  1. 2019SPoC: Search-based Pseudocode to Codecited by 176
  2. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  3. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
  4. 2019On Learning Meaningful Code Changes via Neural Machine Translationcited by 50
  5. 2018Automatic Software Repair: a Bibliographycited by 24
  6. 2020Patching as Translation: the Data and the Metaphorcited by 18
  7. 2020Review4Repair: Code Review Aided Automatic Program Repairingcited by 13
  8. 2014Automatic Repair of Buggy If Conditions and Missing Preconditions with SMTcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 8; 2026: 2.

17 papers.

Sort:

By area:

  1. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  2. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  3. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 22
  4. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  5. 2023INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repaircited by 18
  6. 2024MdEval: Massively Multilingual Code Debuggingcited by 16
  7. 2024CigaR: Cost-efficient Program Repair with LLMscited by 14
  8. 2024COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesiscited by 13
  9. 2025GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Taskscited by 11
  10. 2023RunBugRun -- An Executable Dataset for Automated Program Repaircited by 10
  11. 2025RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Modelscited by 4
  12. 2025Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repaircited by 2
5 more
  1. 2025StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Executioncited by 0
  2. 2025Cerberus: Multi-Agent Reasoning and Coverage-Guided Exploration for Static Detection of Runtime Errorscited by 0
  3. 2025Fuzzwise: Intelligent Initial Corpus Generation for Fuzzingcited by 0
  4. 2026An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinationscited by 0
  5. 2026Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?cited by 0