# ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance

arXiv: https://arxiv.org/abs/2607.02606 · HTML: https://mlsanity.com/paper/2607.02606/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents](https://mlsanity.com/paper/2512.12730/) (2025)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [Repo2Run: Automated Building Executable Environment for Code Repository at Scale](https://mlsanity.com/paper/2502.13681/) (2025)
- [SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale](https://mlsanity.com/paper/2602.23866/) (2026)
- [BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?](https://mlsanity.com/paper/2603.03194/) (2026)
- [When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems](https://mlsanity.com/paper/2606.00448/) (2026)
- [SWE-Edit: Rethinking Code Editing for Efficient SWE-Agent](https://mlsanity.com/paper/2604.26102/) (2026)
- [RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms](https://mlsanity.com/paper/2603.05026/) (2026)
- [ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents](https://mlsanity.com/paper/2604.07789/) (2026)

## Built on by (most cited first, 4 of 4)

- [Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance](https://mlsanity.com/paper/2607.27283/) (2026)
- [StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agents](https://mlsanity.com/paper/2609.36319/) (2026)
- [CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph](https://mlsanity.com/paper/2610.15345/) (2026)
- [StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents](https://mlsanity.com/paper/2610.17616/) (2026)

## Abstract (arXiv)

Language model (LM) agents are increasingly deployed to maintain codebases over extended periods, fixing streams of related defects while carrying context from one fix to the next. Yet existing software engineering (SWE) benchmarks evaluate models one bug at a time: the repository is reset, the codebase is re-read, and a single self-contained issue is graded in isolation. This setting collapses a continuous maintenance workflow into a series of independent sessions, ignoring the cumulative dependencies that make real-world bug fixing challenging. To bridge this gap, we introduce ChainSWE, the first benchmark for evaluating agents on sequential, dependent bug fixes within a shared codebase. We collect chronological chains of 304 issues across 54 Python projects, mined from six SWE-bench-family datasets. Our evaluation across a range of agents and models reveals a consistent performance drop by up to 70% as the chain length increases.