mlsanityData to 14 Oct 2026

SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

arXiv 2512.18470 ↗First posted 20 Dec 2025cs.SE cs.AI cs.MAcited by 59

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 61 of its 80 references that are in the map. First posted on arXiv 20 Dec 2025.

Builds on

61 of 80 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents29 Oct
  2. 2025Huxley-G"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine24 Oct
  3. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents26 May
  4. 2025A Self-Improving Coding Agent21 Apr
  5. 2024Training Software Engineering Agents and Verifiers with SWE-Gym30 Dec
  6. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale9 Sep
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  8. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
  10. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead7 Apr
  11. 2024Executable Code Actions Elicit Better LLM Agents1 Feb

Other references · 50

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  10. 2023Teaching Large Language Models to Self-Debugcited by 988
  11. 2021Measuring Coding Challenge Competence With APPScited by 902
  12. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
38 more
  1. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  2. 2025Kimi K2: Open Agentic Intelligencecited by 567
  3. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  5. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  7. 2022CodeT: Code Generation with Generated Testscited by 384
  8. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  9. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  10. 2023Large Language Models as Tool Makerscited by 303
  11. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  12. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  13. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  14. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  15. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  16. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  17. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
  18. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  19. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  20. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  21. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 116
  22. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  23. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 93
  24. 2025SWE-bench Goes Live!cited by 80
  25. 2024TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Taskscited by 75
  26. 2023A Survey on Large Language Models for Software Engineeringcited by 59
  27. 2024TestGenEval: A Real World Unit Test Generation and Test Completion Benchmarkcited by 52
  28. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 46
  29. 2023Conversational Automated Program Repaircited by 41
  30. 2024TESTEVAL: Benchmarking Large Language Models for Test Case Generationcited by 40
  31. 2025SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugscited by 22
  32. 2023CodeTF: One-stop Transformer Library for State-of-the-art Code LLMscited by 21
  33. 2024The Current Challenges of Software Engineering in the Era of Large Language Modelscited by 21
  34. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  35. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMscited by 18
  36. 2025Context Engineering 2.0: The Context of Context Engineeringcited by 18
  37. 2023Functional Overlap Reranking for Neural Code Generationcited by 9
  38. 2025CodeWiki: Evaluating AI's Ability to Generate Holistic Documentation for Large-Scale Codebasescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 58.

59 papers.

Sort:

By area:

  1. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 59
  2. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 31
  3. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 30
  4. 2026SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integrationcited by 23
  5. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 23
  6. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  7. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  8. 2026SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessionscited by 11
  9. 2026SWE-Milestone: Evaluating AI Agents on Continuous Software Evolutioncited by 8
  10. 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgradescited by 7
  11. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  12. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
38 more
  1. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoringcited by 6
  2. 2026SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?cited by 5
  3. 2026SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Taskscited by 4
  4. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
  5. 2026RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgradescited by 4
  6. 2026Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvementcited by 4
  7. 2026Automated QoR improvement in OpenROAD with coding agentscited by 3
  8. 2026SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agentscited by 3
  9. 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluationcited by 3
  10. 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 3
  11. 2026A Multi-agent AI System for Deep Learning Model Migration from TensorFlow to JAXcited by 2
  12. 2026EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environmentscited by 2
  13. 2026AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?cited by 2
  14. 2026A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agentscited by 2
  15. 2026The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebasecited by 1
  16. 2026Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Editscited by 1
  17. 2026More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystemscited by 1
  18. 2026Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolutioncited by 1
  19. 2026PBT-Bench: Benchmarking AI Agents on Property-Based Testingcited by 1
  20. 2026StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turnscited by 1
  21. 2026PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarkscited by 1
  22. 2026Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environmentscited by 1
  23. 2026Terminal Agents: A Survey of AI Agents in Command-Line Environmentscited by 1
  24. 2026Problem Reductions at Scale: Agentic Integration of Computationally Hard Problemscited by 0
  25. 2026The Conversations Beneath the Code: Triadic Data for Long-Horizon Software Engineering Agentscited by 0
  26. 2026Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performancecited by 0
  27. 2026Shared Organizational Memory for Enterprise Coding Agents: System Design and Deployment Snapshotcited by 0
  28. 2026Persistent Recursive Worlds Enable Autonomous Software Evolutioncited by 0
  29. 2026Evaluating Agentic Code Repair Capabilities in Distributed Systemscited by 0
  30. 2026Repo2Skill-Evo: Repository Skills Go Stale in Silencecited by 0
  31. 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skillscited by 0
  32. 2026Contextual Causality with Large Language Models: A Surveycited by 0
  33. 2026The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Taskscited by 0
  34. 2026SWE-Serve: Benchmarking Agentic Engineering For Production Inference Servingcited by 0
  35. 2026AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineeringcited by 0
  36. 2026Do Coding Agents Reuse Existing Code or Reinvent the Wheel?cited by 0
  37. 2026StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agentscited by 0
  38. 2026MemTrace: State-Consistent Memory for Long-Horizon Coding Agentscited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
  2. 2025SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
  3. 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositories
  4. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
  5. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
  6. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
  7. 2026Self-Evolving Coding Agents
  8. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents