# Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset

arXiv: https://arxiv.org/abs/1811.02429 · HTML: https://mlsanity.com/paper/1811.02429/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2018.

## Built on by (most cited first, 7 of 7)

- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair](https://mlsanity.com/paper/2309.00608/) (2023)
- [Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning](https://mlsanity.com/paper/2301.01113/) (2023)
- [Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems](https://mlsanity.com/paper/2506.17208/) (2025)
- [Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency](https://mlsanity.com/paper/2608.14065/) (2026)
- [Exploring Plausible Patches Using Source Code Embeddings in JavaScript](https://mlsanity.com/paper/2103.16846/) (2021)
- [Mapping the Structure and Evolution of Software Testing Research Over the Past Three Decades](https://mlsanity.com/paper/2109.04086/) (2021)

## Abstract (arXiv)

Defects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J comes with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic test-suite based repair on Defects4J. The result of our experiment shows that the considered state-of-the-art repair methods can generate patches for 47 out of 224 bugs. However, those patches are only test-suite adequate, which means that they pass the test suite and may potentially be incorrect beyond the test-suite satisfaction correctness criterion. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly repaired with test-suite based repair. This analysis shows that test-suite based repair suffers from under-specified bugs, for which trivial or incorrect patches still pass the test suite. With respect to practical applicability, it takes on average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of computation time. All the repair systems and experimental results are publicly available on Github in order to facilitate future research on automatic repair.