# Challenging Bug Prediction and Repair Models with Synthetic Bugs

arXiv: https://arxiv.org/abs/2310.02407 · HTML: https://mlsanity.com/paper/2310.02407/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [In-context Example Selection with Influences](https://mlsanity.com/paper/2302.11042/) (2023)
- [NatGen: Generative pre-training by "Naturalizing" source code](https://mlsanity.com/paper/2206.07585/) (2022)
- [Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond](https://mlsanity.com/paper/2304.05216/) (2023)
- [Learning How to Mutate Source Code from Bug-Fixes](https://mlsanity.com/paper/1812.10772/) (2018)
- [BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes](https://mlsanity.com/paper/1903.06725/) (2019)
- [Revisiting the Plastic Surgery Hypothesis via Large Language Models](https://mlsanity.com/paper/2303.10494/) (2023)
- [Perfect is the enemy of test oracle](https://mlsanity.com/paper/2302.01488/) (2023)
- [Efficient Mutation Testing via Pre-Trained Language Models](https://mlsanity.com/paper/2301.03543/) (2023)
- [RegMiner: Towards Constructing a Large Regression Dataset from Code Evolution History](https://mlsanity.com/paper/2109.12389/) (2021)
- [What It Would Take to Use Mutation Testing in Industry--A Study at Facebook](https://mlsanity.com/paper/2010.13464/) (2020)

## Built on by (most cited first, 4 of 4)

- [SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?](https://mlsanity.com/paper/2605.22175/) (2026)
- [CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks](https://mlsanity.com/paper/2507.05281/) (2025)
- [Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking](https://mlsanity.com/paper/2606.29088/) (2026)
- [Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models](https://mlsanity.com/paper/2609.35841/) (2026)

## Abstract (arXiv)

Bugs are essential in software engineering; many research studies in the past decades have been proposed to detect, localize, and repair bugs in software systems. Effectiveness evaluation of such techniques requires complex bugs, i.e., those that are hard to detect through testing and hard to repair through debugging. From the classic software engineering point of view, a hard-to-repair bug differs from the correct code in multiple locations, making it hard to localize and repair. Hard-to-detect bugs, on the other hand, manifest themselves under specific test inputs and reachability conditions. These two objectives, i.e., generating hard-to-detect and hard-to-repair bugs, are mostly aligned; a bug generation technique can change multiple statements to be covered only under a specific set of inputs. However, these two objectives are conflicting for learning-based techniques: A bug should have a similar code representation to the correct code in the training data to challenge a bug prediction model to distinguish them. The hard-to-repair bug definition remains the same but with a caveat: the more a bug differs from the original code, the more distant their representations are and easier to be detected. We propose BugFarm, to transform arbitrary code into multiple complex bugs. BugFarm leverages LLMs to mutate code in multiple locations (hard-to-repair). To ensure that multiple modifications do not notably change the code representation, BugFarm analyzes the attention of the underlying model and instructs LLMs to only change the least attended locations (hard-to-detect). Our comprehensive evaluation of 435k+ bugs from over 1.9M mutants generated by BUGFARM and two alternative approaches demonstrates our superiority in generating bugs that are hard to detect by learning-based bug prediction approaches and hard-to-repair by state-of-the-art learning-based program repair technique.