# TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

arXiv: https://arxiv.org/abs/2607.02469 · HTML: https://mlsanity.com/paper/2607.02469/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [Automated Test Case Repair Using Language Models](https://mlsanity.com/paper/2401.06765/) (2024)
- [Fix the Tests: Augmenting LLMs to Repair Test Cases with Static Collector and Neural Reranker](https://mlsanity.com/paper/2407.03625/) (2024)
- [UTFix: Change Aware Unit Test Repairing using LLM](https://mlsanity.com/paper/2503.14924/) (2025)
- [Unit Test Update through LLM-Driven Context Collection and Error-Type-Aware Refinement](https://mlsanity.com/paper/2509.24419/) (2025)
- [The Co-Evolution of Test Maintenance and Code Maintenance through the lens of Fine-Grained Semantic Changes](https://mlsanity.com/paper/1709.09029/) (2017)
- [LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework](https://mlsanity.com/paper/2511.20403/) (2025)

## Built on by (most cited first, 1 of 1)

- [Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments](https://mlsanity.com/paper/2607.28591/) (2026)

## Abstract (arXiv)

Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.