# SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

arXiv: https://arxiv.org/abs/2512.18470 · HTML: https://mlsanity.com/paper/2512.18470/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 61 of its 80 references that are in the map.
- First posted on arXiv 20 Dec 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 61)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025)

## Built on by (most cited first, 30 of 59)

- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading](https://mlsanity.com/paper/2607.08964/) (2026)
- [SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks](https://mlsanity.com/paper/2603.24755/) (2026)
- [SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration](https://mlsanity.com/paper/2603.03823/) (2026)
- [SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?](https://mlsanity.com/paper/2606.07682/) (2026)
- [Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned](https://mlsanity.com/paper/2603.05344/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions](https://mlsanity.com/paper/2606.30573/) (2026)
- [SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution](https://mlsanity.com/paper/2603.13428/) (2026)
- [SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades](https://mlsanity.com/paper/2605.14415/) (2026)
- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://mlsanity.com/paper/2608.09802/) (2026)
- [SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?](https://mlsanity.com/paper/2608.23564/) (2026)
- [SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks](https://mlsanity.com/paper/2603.00575/) (2026)
- [VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?](https://mlsanity.com/paper/2605.06068/) (2026)
- [RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades](https://mlsanity.com/paper/2605.15846/) (2026)
- [Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement](https://mlsanity.com/paper/2609.01481/) (2026)
- [Automated QoR improvement in OpenROAD with coding agents](https://mlsanity.com/paper/2601.06268/) (2026)
- [SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents](https://mlsanity.com/paper/2602.09447/) (2026)
- [LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation](https://mlsanity.com/paper/2608.00267/) (2026)
- [SWE-Touch: Benchmarking Coding Agents When Users Touch the Code](https://mlsanity.com/paper/2608.02499/) (2026)
- [A Multi-agent AI System for Deep Learning Model Migration from TensorFlow to JAX](https://mlsanity.com/paper/2603.27296/) (2026)
- [EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments](https://mlsanity.com/paper/2607.02440/) (2026)
- [AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?](https://mlsanity.com/paper/2607.03332/) (2026)
- [A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents](https://mlsanity.com/paper/2608.09072/) (2026)
- [The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebase](https://mlsanity.com/paper/2603.25697/) (2026)
- [Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits](https://mlsanity.com/paper/2603.27745/) (2026)
- [More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystems](https://mlsanity.com/paper/2604.19827/) (2026)
- [Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution](https://mlsanity.com/paper/2605.06125/) (2026)

## Abstract (arXiv)

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.