# Issue2Test: Generating Reproducing Test Cases from Issue Reports

arXiv: https://arxiv.org/abs/2503.16320 · HTML: https://mlsanity.com/paper/2503.16320/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2025.

## Built on by (most cited first, 18 of 18)

- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [AgentStepper: Interactive Debugging of Software Development Agents](https://mlsanity.com/paper/2602.06593/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation](https://mlsanity.com/paper/2602.10471/) (2026)
- [SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests](https://mlsanity.com/paper/2607.00990/) (2026)
- [SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories](https://mlsanity.com/paper/2601.13713/) (2026)
- [Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes](https://mlsanity.com/paper/2607.19843/) (2026)
- [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026)
- [Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair](https://mlsanity.com/paper/2511.11012/) (2025)
- [EvoOtter: Evolutionary Reproduction Test Generator](https://mlsanity.com/paper/2607.02854/) (2026)
- [ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports](https://mlsanity.com/paper/2607.09123/) (2026)
- [From Benchmark Data To Applicable Program Repair: An Experience Report](https://mlsanity.com/paper/2508.16071/) (2025)
- [SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning](https://mlsanity.com/paper/2601.21649/) (2026)
- [Evaluating LLM Agents on Automated Software Analysis Tasks](https://mlsanity.com/paper/2604.11270/) (2026)
- [Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?](https://mlsanity.com/paper/2607.28871/) (2026)
- [DPIAgent: Divide, Protocol, Isolate for Agentic Reproduction Test Generation](https://mlsanity.com/paper/2608.23341/) (2026)
- [XAgent: eXecution-guided Agentic AI for Effective Localization and Resolution of GitHub Issues](https://mlsanity.com/paper/2609.09769/) (2026)
- [WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses](https://mlsanity.com/paper/2609.36635/) (2026)

## Abstract (arXiv)

Automated tools for solving GitHub issues are receiving significant attention by both researchers and practitioners, e.g., in the form of foundation models and LLM-based agents prompted with issues. A crucial step toward successfully solving an issue is creating a test case that accurately reproduces the issue. Such a test case can guide the search for an appropriate patch and help validate whether the patch matches the issue's intent. However, existing techniques for issue reproduction show only moderate success. This paper presents Issue2Test, an LLM-based technique for automatically generating a reproducing test case for a given issue report. Unlike automated regression test generators, which aim at creating passing tests, our approach aims at a test that fails, and that fails specifically for the reason described in the issue. To this end, Issue2Test performs three steps: (1) understand the issue and gather context (e.g., related files and project-specific guidelines) relevant for reproducing it; (2) generate a candidate test case; and (3) iteratively refine the test case based on compilation and runtime feedback until it fails and the failure aligns with the problem described in the issue. We evaluate Issue2Test on the SWT-bench-lite dataset, where it successfully reproduces 32.9% of the issues, achieving a 16.3% relative improvement over the best existing technique. Our evaluation also shows that Issue2Test reproduces 20 issues that four prior techniques fail to address, contributing a total of 60.4% of all issues reproduced by these tools. We envision our approach to contribute to enhancing the overall progress in the important task of automatically solving GitHub issues.