# EvoOtter: Evolutionary Reproduction Test Generator

arXiv: https://arxiv.org/abs/2607.02854 · HTML: https://mlsanity.com/paper/2607.02854/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning](https://mlsanity.com/paper/2506.03136/) (2025)
- [CodeMonkeys: Scaling Test-Time Compute for Software Engineering](https://mlsanity.com/paper/2501.14723/) (2025)
- [The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason](https://mlsanity.com/paper/2506.12286/) (2025)
- [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024)
- [Otter: Generating Tests from Issues to Validate SWE Patches](https://mlsanity.com/paper/2502.05368/) (2025)
- [Agentic Bug Reproduction for Effective Automated Program Repair at Google](https://mlsanity.com/paper/2502.01821/) (2025)
- [Issue2Test: Generating Reproducing Test Cases from Issue Reports](https://mlsanity.com/paper/2503.16320/) (2025)
- [AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions](https://mlsanity.com/paper/2411.18015/) (2024)
- [Selecting Near-Optimal Learners via Incremental Data Allocation](https://mlsanity.com/paper/1601.00024/) (2015)
- [AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests](https://mlsanity.com/paper/2507.17542/) (2025)
- [Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection](https://mlsanity.com/paper/2508.06365/) (2025)
- [Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems](https://mlsanity.com/paper/2506.17208/) (2025)
- [REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement](https://mlsanity.com/paper/2510.03588/) (2025)
- [iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation](https://mlsanity.com/paper/2604.19224/) (2026)

## Built on by (most cited first, 1 of 1)

- [Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?](https://mlsanity.com/paper/2607.28871/) (2026)

## Abstract (arXiv)

Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using large language models to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with large language models for software engineering.