# GitBug-Java: A Reproducible Benchmark of Recent Java Bugs

arXiv: https://arxiv.org/abs/2402.02961 · HTML: https://mlsanity.com/paper/2402.02961/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2024.

## Built on by (most cited first, 14 of 14)

- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks](https://mlsanity.com/paper/2502.06215/) (2025)
- [Are Large Language Models Memorizing Bug Benchmarks?](https://mlsanity.com/paper/2411.13323/) (2024)
- [RepairBench: Leaderboard of Frontier Models for Program Repair](https://mlsanity.com/paper/2409.18952/) (2024)
- [Improving Deep Assertion Generation via Fine-Tuning Retrieval-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2502.16071/) (2025)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [Improving Retrieval-Augmented Deep Assertion Generation via Joint Training](https://mlsanity.com/paper/2502.10696/) (2025)
- [Are Sparse Autoencoders Useful for Java Function Bug Detection?](https://mlsanity.com/paper/2505.10375/) (2025)
- [PatchBench: Evaluating AI Agents for Vulnerability Patching](https://mlsanity.com/paper/2609.04075/) (2026)
- [Debug2Fix: Can Interactive Debugging Help Coding Agents Fix More Bugs?](https://mlsanity.com/paper/2602.18571/) (2026)
- [A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair](https://mlsanity.com/paper/2604.21579/) (2026)
- [Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)](https://mlsanity.com/paper/2607.22880/) (2026)
- [Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk](https://mlsanity.com/paper/2610.14460/) (2026)

## Abstract (arXiv)

Bug-fix benchmarks are essential for evaluating methodologies in automatic program repair (APR) and fault localization (FL). However, existing benchmarks, exemplified by Defects4J, need to evolve to incorporate recent bug-fixes aligned with contemporary development practices. Moreover, reproducibility, a key scientific principle, has been lacking in bug-fix benchmarks. To address these gaps, we present GitBug-Java, a reproducible benchmark of recent Java bugs. GitBug-Java features 199 bugs extracted from the 2023 commit history of 55 notable open-source repositories. The methodology for building GitBug-Java ensures the preservation of bug-fixes in fully-reproducible environments. We publish GitBug-Java at https://github.com/gitbugactions/gitbug-java.