# Investigating Test Overfitting on SWE-bench

arXiv: https://arxiv.org/abs/2511.16858 · HTML: https://mlsanity.com/paper/2511.16858/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [S*: Test Time Scaling for Code Generation](https://mlsanity.com/paper/2502.14382/) (2025)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [CodeR: Issue Resolving with Multi-Agent and Task Graphs](https://mlsanity.com/paper/2406.01304/) (2024)
- [SpecRover: Code Intent Extraction via LLMs](https://mlsanity.com/paper/2408.02232/) (2024)
- [CodeMonkeys: Scaling Test-Time Compute for Software Engineering](https://mlsanity.com/paper/2501.14723/) (2025)
- [MASAI: Modular Architecture for Software-engineering AI Agents](https://mlsanity.com/paper/2406.11638/) (2024)
- [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024)
- [Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection](https://mlsanity.com/paper/2508.06365/) (2025)
- [Can Old Tests Do New Tricks for Resolving SWE Issues?](https://mlsanity.com/paper/2510.18270/) (2025)

## Built on by (most cited first, 6 of 6)

- [From Plan to Action: How Well Do Agents Follow the Plan?](https://mlsanity.com/paper/2604.12147/) (2026)
- [ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositories](https://mlsanity.com/paper/2604.07341/) (2026)
- [SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution](https://mlsanity.com/paper/2608.18933/) (2026)
- [Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?](https://mlsanity.com/paper/2609.27891/) (2026)
- [Teaching Agents to Code Reliably](https://mlsanity.com/paper/2610.03984/) (2026)
- [SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?](https://mlsanity.com/paper/2610.16870/) (2026)

## Abstract (arXiv)

Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests but actually misses important cases or even breaks functionality. This problem, called test overfitting, is exacerbated by the fact that issues usually lack readily executable tests. Instead, several issue resolution systems use tests auto-generated from issues, which may be imperfect. Some systems even iteratively refine code and tests jointly. This paper presents the first empirical study of test overfitting in this setting.