mlsanityData to 14 Oct 2026

Investigating Test Overfitting on SWE-bench

arXiv 2511.16858 ↗First posted 20 Nov 2025cs.SE cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests but actually misses important cases or even breaks functionality. This problem, called test overfitting, is exacerbated by the fact that issues usually lack readily executable tests. Instead, several issue resolution systems use tests auto-generated from issues, which may be imperfect. Some systems even iteratively refine code and tests jointly. This paper presents the first empirical study of test overfitting in this setting.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

14 of 28 references.

References · 14

  1. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  2. 2022CodeT: Code Generation with Generated Testscited by 384
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  4. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  5. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 133
  6. 2025S*: Test Time Scaling for Code Generationcited by 98
  7. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 94
  8. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
  9. 2024SpecRover: Code Intent Extraction via LLMscited by 72
  10. 2025CodeMonkeys: Scaling Test-Time Compute for Software Engineeringcited by 51
  11. 2024MASAI: Modular Architecture for Software-engineering AI Agentscited by 49
  12. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 34
2 more
  1. 2025Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selectioncited by 10
  2. 2025Can Old Tests Do New Tricks for Resolving SWE Issues?cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026From Plan to Action: How Well Do Agents Follow the Plan?cited by 4
  2. 2026ReCodeAgent: A Multi-agent Workflow for Language-Agnostic Translation and Validation of Large-Scale Repositoriescited by 2
  3. 2026SkillForge: Self-Distilling Agents for Project-Specific Issue Resolutioncited by 2
  4. 2026Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?cited by 0
  5. 2026Teaching Agents to Code Reliablycited by 0
  6. 2026SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?cited by 0