# Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

arXiv: https://arxiv.org/abs/2602.07900 · HTML: https://mlsanity.com/paper/2602.07900/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 14 of its 59 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 14 of 14)

- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Towards a Science of Scaling Agent Systems](https://mlsanity.com/paper/2512.08296/) (2025)
- [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024)
- [CodeMonkeys: Scaling Test-Time Compute for Software Engineering](https://mlsanity.com/paper/2501.14723/) (2025)
- [EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair](https://mlsanity.com/paper/2506.10484/) (2025)
- [Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study](https://mlsanity.com/paper/2506.08311/) (2025)
- [An Empirical Study on Failures in Automated Issue Solving](https://mlsanity.com/paper/2509.13941/) (2025)

## Built on by (most cited first, 14 of 14)

- [Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures](https://mlsanity.com/paper/2604.03515/) (2026)
- [Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure](https://mlsanity.com/paper/2604.02547/) (2026)
- [EffiSkill: Agent Skill Based Automated Code Efficiency Optimization](https://mlsanity.com/paper/2603.27850/) (2026)
- [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026)
- [Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?](https://mlsanity.com/paper/2607.01211/) (2026)
- [Beyond Code Generation: Reliability, Verification, and Cost Economics in the Agentic Software Development Lifecycle](https://mlsanity.com/paper/2609.04681/) (2026)
- [Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents](https://mlsanity.com/paper/2610.01023/) (2026)
- [Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?](https://mlsanity.com/paper/2607.28871/) (2026)
- [Towards Assurance Closure in AI-Native Large-Scale Agile Software Development](https://mlsanity.com/paper/2608.07317/) (2026)
- [Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?](https://mlsanity.com/paper/2609.27891/) (2026)
- [Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification](https://mlsanity.com/paper/2609.38812/) (2026)
- [Rules to Tools: Executable Checks for LLM Agents in Scientific Computing](https://mlsanity.com/paper/2610.00313/) (2026)
- [When Algorithmic Exploration Becomes Cheap: A Case Study of Agentic Research in EDA](https://mlsanity.com/paper/2610.10129/) (2026)
- [DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck](https://mlsanity.com/paper/2610.17184/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.