# BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies

arXiv: https://arxiv.org/abs/2401.15481 · HTML: https://mlsanity.com/paper/2401.15481/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2024.

## Built on by (most cited first, 30 of 36)

- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024)
- [HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale](https://mlsanity.com/paper/2409.16299/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [CoverUp: Effective High Coverage Test Generation for Python](https://mlsanity.com/paper/2403.16218/) (2024)
- [LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks](https://mlsanity.com/paper/2502.06215/) (2025)
- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)
- [Are Large Language Models Memorizing Bug Benchmarks?](https://mlsanity.com/paper/2411.13323/) (2024)
- [Measuring the Influence of Incorrect Code on Test Generation](https://mlsanity.com/paper/2409.09464/) (2024)
- [Corrective Diffusion Language Models](https://mlsanity.com/paper/2512.15596/) (2025)
- [SERA: Soft-Verified Efficient Repository Agents](https://mlsanity.com/paper/2601.20789/) (2026)
- [A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Tasks](https://mlsanity.com/paper/2511.00872/) (2025)
- [Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing](https://mlsanity.com/paper/2402.11892/) (2024)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [Assessing the Impact of Code Changes on the Fault Localizability of Large Language Models](https://mlsanity.com/paper/2504.04372/) (2025)
- [Improving LLM Reasoning for Vulnerability Detection via Group Relative Policy Optimization](https://mlsanity.com/paper/2507.03051/) (2025)
- [When Names Disappear: Revealing What LLMs Actually Understand About Code](https://mlsanity.com/paper/2510.03178/) (2025)
- [OmniCode: A Benchmark for Evaluating Software Engineering Agents](https://mlsanity.com/paper/2602.02262/) (2026)
- [Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization](https://mlsanity.com/paper/2412.18750/) (2024)
- [Fault Localization via Fine-tuning Large Language Models with Mutation Generated Stack Traces](https://mlsanity.com/paper/2501.18005/) (2025)
- [PyResBugs: A Dataset of Residual Python Bugs for Natural Language-Driven Fault Injection](https://mlsanity.com/paper/2505.05777/) (2025)
- [From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs](https://mlsanity.com/paper/2506.13182/) (2025)
- [PBT-Bench: Benchmarking AI Agents on Property-Based Testing](https://mlsanity.com/paper/2605.15229/) (2026)
- [Anchored Self-Play for Code Repair](https://mlsanity.com/paper/2607.03523/) (2026)
- [SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents](https://mlsanity.com/paper/2609.04167/) (2026)
- [SLICEMATE: Accurate and Scalable Static Program Slicing via LLM-Powered Agents](https://mlsanity.com/paper/2507.18957/) (2025)
- [Tricky2: Towards a Benchmark for Evaluating Human and LLM Error Interactions](https://mlsanity.com/paper/2601.18949/) (2026)
- [SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization](https://mlsanity.com/paper/2603.29109/) (2026)
- [An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code](https://mlsanity.com/paper/2604.23361/) (2026)
- [All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application](https://mlsanity.com/paper/2606.22475/) (2026)

## Abstract (arXiv)

The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers. One factor that contributed to the rapid growth of Java testing and debugging tools is the availability of benchmarks. A popular benchmark is the Defects4J benchmark; its initial version contained 357 real bugs from 5 real-world Java programs. Each bug comes with a test suite that can expose the bug. Defects4J has been used by hundreds of testing and debugging studies and has helped to push the frontier of research in these directions. In this project, inspired by Defects4J, we create another benchmark database and tool that contain 493 real bugs from 17 real-world Python programs. We hope our benchmark can help catalyze future work on testing and debugging tools that work on Python programs.