# RunBugRun -- An Executable Dataset for Automated Program Repair

arXiv: https://arxiv.org/abs/2304.01102 · HTML: https://mlsanity.com/paper/2304.01102/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 87 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 30 of 41)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation](https://mlsanity.com/paper/1812.08693/) (2018)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [Self-Supervised Bug Detection and Repair](https://mlsanity.com/paper/2105.12787/) (2021)
- [Neural Program Repair with Execution-based Backpropagation](https://mlsanity.com/paper/2105.04123/) (2021)
- [Multilingual training for Software Engineering](https://mlsanity.com/paper/2112.02043/) (2021)
- [Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis](https://mlsanity.com/paper/2007.08095/) (2020)
- [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming](https://mlsanity.com/paper/1712.07804/) (2017)
- [Dynamic Neural Program Embedding for Program Repair](https://mlsanity.com/paper/1711.07163/) (2017)
- [A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark](https://mlsanity.com/paper/1805.03454/) (2018)
- [Identifying Patch Correctness in Test-Based Program Repair](https://mlsanity.com/paper/1706.09120/) (2017)
- [DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons](https://mlsanity.com/paper/2105.09352/) (2021)
- [FixEval: Execution-based Evaluation of Program Fixes for Programming Problems](https://mlsanity.com/paper/2206.07796/) (2022)
- [Elixir: Effective object-oriented program repair](https://mlsanity.com/paper/2112.10915/) (2021)
- [On Distribution Shift in Learning-based Bug Detectors](https://mlsanity.com/paper/2204.10049/) (2022)
- [Automated Patch Assessment for Program Repair at Scale](https://mlsanity.com/paper/1909.13694/) (2019)
- [TSSB-3M: Mining single statement bugs at massive scale](https://mlsanity.com/paper/2201.12046/) (2022)

## Built on by (most cited first, 11 of 11)

- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [IFEvalCode: Controlled Code Generation](https://mlsanity.com/paper/2507.22462/) (2025)
- [Aletheia: What Makes RLVR For Code Verifiers Tick?](https://mlsanity.com/paper/2601.12186/) (2026)
- [Multilingual Multimodal Software Developer for Code Generation](https://mlsanity.com/paper/2507.08719/) (2025)
- [ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptions](https://mlsanity.com/paper/2503.04241/) (2025)
- [Codehacks: A Dataset of Adversarial Tests for Competitive Programming Problems Obtained from Codeforces](https://mlsanity.com/paper/2503.23466/) (2025)
- [Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations](https://mlsanity.com/paper/2607.01181/) (2026)
- [Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring](https://mlsanity.com/paper/2605.00754/) (2026)
- [How Much Evidence Should a Coding Agent's Self-Correction Carry? Adaptive Dirichlet Evidence for Self-Distillation](https://mlsanity.com/paper/2610.08514/) (2026)

## Abstract (arXiv)

Recently, we can notice a transition to data-driven techniques in Automated Program Repair (APR), in particular towards deep neural networks. This entails training on hundreds of thousands or even millions of non-executable code fragments. We would like to bring more attention to an aspect of code often neglected in Neural Program Repair (NPR), namely its execution. Code execution has several significant advantages. It allows for test-based evaluation of candidate fixes and can provide valuable information to aid repair. In this work we present a fully executable dataset of 450,000 small buggy/fixed program pairs originally submitted to programming competition websites written in eight different programming languages. Along with the dataset we provide infrastructure to compile, safely execute and test programs as well as fine-grained bug-type labels. To give a point of reference, we provide basic evaluation results for two baselines, one based on a generate-and-validate approach and one on deep learning. With this dataset we follow several goals: we want to lift Neural Program Repair beyond fully static code representations, foster the use of execution-based features and, by including several different languages, counterbalance the predominance of Java in the current landscape of APR datasets and benchmarks.