# FixEval: Execution-based Evaluation of Program Fixes for Programming Problems

arXiv: https://arxiv.org/abs/2206.07796 · HTML: https://mlsanity.com/paper/2206.07796/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [AVATAR: A Parallel Corpus for Java-Python Program Translation](https://mlsanity.com/paper/2108.11590/) (2021)
- [On Learning Meaningful Code Changes via Neural Machine Translation](https://mlsanity.com/paper/1901.09102/) (2019)
- [Automatic Software Repair: a Bibliography](https://mlsanity.com/paper/1807.00515/) (2018)
- [Patching as Translation: the Data and the Metaphor](https://mlsanity.com/paper/2008.10707/) (2020)
- [Review4Repair: Code Review Aided Automatic Program Repairing](https://mlsanity.com/paper/2010.01544/) (2020)
- [Automatic Repair of Buggy If Conditions and Missing Preconditions with SMT](https://mlsanity.com/paper/1404.3186/) (2014)

## Built on by (most cited first, 17 of 17)

- [OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs](https://mlsanity.com/paper/2504.04030/) (2025)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [CodeEditorBench: Evaluating Code Editing Capability of Large Language Models](https://mlsanity.com/paper/2404.03543/) (2024)
- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)
- [INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair](https://mlsanity.com/paper/2311.09868/) (2023)
- [MdEval: Massively Multilingual Code Debugging](https://mlsanity.com/paper/2411.02310/) (2024)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)
- [GeoBenchX: Benchmarking LLMs in Agent Solving Multistep Geospatial Tasks](https://mlsanity.com/paper/2503.18129/) (2025)
- [COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis](https://mlsanity.com/paper/2408.05006/) (2024)
- [RunBugRun -- An Executable Dataset for Automated Program Repair](https://mlsanity.com/paper/2304.01102/) (2023)
- [RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models](https://mlsanity.com/paper/2509.04078/) (2025)
- [Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair](https://mlsanity.com/paper/2512.22633/) (2025)
- [StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Execution](https://mlsanity.com/paper/2508.11665/) (2025)
- [Cerberus: Multi-Agent Reasoning and Coverage-Guided Exploration for Static Detection of Runtime Errors](https://mlsanity.com/paper/2512.21431/) (2025)
- [Fuzzwise: Intelligent Initial Corpus Generation for Fuzzing](https://mlsanity.com/paper/2512.21440/) (2025)
- [An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations](https://mlsanity.com/paper/2604.07755/) (2026)
- [Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?](https://mlsanity.com/paper/2609.29410/) (2026)

## Abstract (arXiv)

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs. In response, researchers have explored various methods to automatically generate fixes for buggy code. However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively. To address this issue, we introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes. FixEval offers an extensive collection of unit tests to evaluate the correctness of model-generated program fixes and assess further information regarding time, memory constraints, and acceptance based on a verdict. We consider two Transformer language models pretrained on programming languages as our baseline and compare them using match-based and execution-based evaluation metrics. Our experiments show that match-based metrics do not reflect model-generated program fixes accurately. At the same time, execution-based methods evaluate programs through all cases and scenarios designed explicitly for that solution. Therefore, we believe FixEval provides a step towards real-world automatic bug fixing and model-generated code evaluation. The dataset and models are open-sourced at https://github.com/mahimanzum/FixEval.