# BigIssue: A Realistic Bug Localization Benchmark

arXiv: https://arxiv.org/abs/2207.10739 · HTML: https://mlsanity.com/paper/2207.10739/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [Graph-based, Self-Supervised Program Repair from Diagnostic Feedback](https://mlsanity.com/paper/2005.10636/) (2020)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019)
- [DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons](https://mlsanity.com/paper/2105.09352/) (2021)
- [CIRCLE: Continual Repair across Programming Languages](https://mlsanity.com/paper/2205.10956/) (2022)
- [An Empirical Study of Fault Localization Families and Their Combinations](https://mlsanity.com/paper/1803.09939/) (2018)
- [You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems](https://mlsanity.com/paper/1812.07283/) (2018)
- [Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts](https://mlsanity.com/paper/1905.11973/) (2019)
- [FLACOCO: Fault Localization for Java based on Industry-grade Coverage](https://mlsanity.com/paper/2111.12513/) (2021)
- [Does Unit-Tested Code Crash? A Case Study of Eclipse](https://mlsanity.com/paper/1903.04055/) (2019)
- [Obstacles in Fully Automatic Program Repair: A survey](https://mlsanity.com/paper/2011.02714/) (2020)

## Abstract (arXiv)

As machine learning tools progress, the inevitable question arises: How can machine learning help us write better code? With significant progress being achieved in natural language processing with models like GPT-3 and Bert, the applications of natural language processing techniques to code are starting to be explored. Most of the research has been focused on automatic program repair (APR), and while the results on synthetic or highly filtered datasets are promising, such models are hard to apply in real-world scenarios because of inadequate bug localization. We propose BigIssue: a benchmark for realistic bug localization. The goal of the benchmark is two-fold. We provide (1) a general benchmark with a diversity of real and synthetic Java bugs and (2) a motivation to improve bug localization capabilities of models through attention to the full repository context. With the introduction of BigIssue, we hope to advance the state of the art in bug localization, in turn improving APR performance and increasing its applicability to the modern development cycle.