# BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes

arXiv: https://arxiv.org/abs/1903.06725 · HTML: https://mlsanity.com/paper/1903.06725/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2019.

## Built on by (most cited first, 12 of 12)

- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs](https://mlsanity.com/paper/2408.02193/) (2024)
- [RunBugRun -- An Executable Dataset for Automated Program Repair](https://mlsanity.com/paper/2304.01102/) (2023)
- [DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle](https://mlsanity.com/paper/2601.20882/) (2026)
- [Challenging Bug Prediction and Repair Models with Synthetic Bugs](https://mlsanity.com/paper/2310.02407/) (2023)
- [FuzzAug: Data Augmentation by Coverage-guided Fuzzing for Neural Test Generation](https://mlsanity.com/paper/2406.08665/) (2024)
- [From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs](https://mlsanity.com/paper/2506.13182/) (2025)
- [Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation](https://mlsanity.com/paper/2512.01255/) (2025)
- [Infrared: A Meta Bug Detector](https://mlsanity.com/paper/2209.08510/) (2022)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems](https://mlsanity.com/paper/2609.37864/) (2026)

## Abstract (arXiv)

Fault-detection, localization, and repair methods are vital to software quality; but it is difficult to evaluate their generality, applicability, and current effectiveness. Large, diverse, realistic datasets of durably-reproducible faults and fixes are vital to good experimental evaluation of approaches to software quality, but they are difficult and expensive to assemble and keep current. Modern continuous-integration (CI) approaches, like Travis-CI, which are widely used, fully configurable, and executed within custom-built containers, promise a path toward much larger defect datasets. If we can identify and archive failing and subsequent passing runs, the containers will provide a substantial assurance of durable future reproducibility of build and test. Several obstacles, however, must be overcome to make this a practical reality. We describe BugSwarm, a toolset that navigates these obstacles to enable the creation of a scalable, diverse, realistic, continuously growing set of durably reproducible failing and passing versions of real-world, open-source systems. The BugSwarm toolkit has already gathered 3,091 fail-pass pairs, in Java and Python, all packaged within fully reproducible containers. Furthermore, the toolkit can be run periodically to detect fail-pass activities, thus growing the dataset continually.