# SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?

arXiv: https://arxiv.org/abs/2606.07682 · HTML: https://mlsanity.com/paper/2606.07682/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 73 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Learning to Discover at Test Time](https://mlsanity.com/paper/2601.16175/) (2026)
- [SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios](https://mlsanity.com/paper/2512.18470/) (2025)
- [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021)
- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges](https://mlsanity.com/paper/2604.13602/) (2026)
- [Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use](https://mlsanity.com/paper/2605.02964/) (2026)
- [LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking](https://mlsanity.com/paper/2604.15149/) (2026)
- [SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents](https://mlsanity.com/paper/2605.21384/) (2026)
- [Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis](https://mlsanity.com/paper/2601.20103/) (2026)
- [Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR](https://mlsanity.com/paper/2603.07084/) (2026)
- [Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories](https://mlsanity.com/paper/2604.17596/) (2026)
- [Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale](https://mlsanity.com/paper/2605.20744/) (2026)
- [Towards Understanding Specification Gaming in Reasoning Models](https://mlsanity.com/paper/2605.02269/) (2026)
- [Technical Debt Prioritization: State of the Art. A Systematic Literature Review](https://mlsanity.com/paper/1904.12538/) (2019)

## Built on by (most cited first, 23 of 23)

- [Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading](https://mlsanity.com/paper/2607.08964/) (2026)
- [Recursive Synthesis for Long-Horizon Terminal Tasks](https://mlsanity.com/paper/2608.05466/) (2026)
- [MirrorCode: AI can rebuild entire programs from behavior alone](https://mlsanity.com/paper/2606.30182/) (2026)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://mlsanity.com/paper/2608.09802/) (2026)
- [BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks](https://mlsanity.com/paper/2608.25286/) (2026)
- [SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents](https://mlsanity.com/paper/2609.08149/) (2026)
- [Beyond Code Generation: Reliability, Verification, and Cost Economics in the Agentic Software Development Lifecycle](https://mlsanity.com/paper/2609.04681/) (2026)
- [Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance](https://mlsanity.com/paper/2607.27283/) (2026)
- [BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests](https://mlsanity.com/paper/2608.02685/) (2026)
- [Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Traces](https://mlsanity.com/paper/2609.20844/) (2026)
- [SWE-Game: Can Coding Agents Build the Games We Want?](https://mlsanity.com/paper/2609.33678/) (2026)
- [Marathoner: Ultra-Long-Horizon Autonomous Intelligence](https://mlsanity.com/paper/2609.34378/) (2026)
- [StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agents](https://mlsanity.com/paper/2609.36319/) (2026)
- [TomasuLLM: Out-of-Order Speculative Execution for LLM Agents](https://mlsanity.com/paper/2609.38201/) (2026)
- [Incident-Arena: Getting agents to the last nine of reliability](https://mlsanity.com/paper/2610.00648/) (2026)
- [Cross-Benchmark Transfer from RL on Agentic Coding Tasks](https://mlsanity.com/paper/2610.00890/) (2026)
- [Does an Agent's History Tell You When Compaction Will Hurt? A Modest, Bounded Effect on the TRACE Paired-Replay Corpus](https://mlsanity.com/paper/2610.08722/) (2026)
- [LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets](https://mlsanity.com/paper/2610.09872/) (2026)
- [TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution](https://mlsanity.com/paper/2610.10619/) (2026)
- [SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction](https://mlsanity.com/paper/2610.11559/) (2026)
- [Scaling Agentic Data for Long-Horizon Terminal Intelligence](https://mlsanity.com/paper/2610.13166/) (2026)
- [WorkBorne: Discovering Agentic Tasks from Real-World Artifact Transitions](https://mlsanity.com/paper/2610.14915/) (2026)
- [EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications](https://mlsanity.com/paper/2610.16402/) (2026)

## Abstract (arXiv)

AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form tasks, such as single pull requests, small tickets, or 5-10 minute exercises, limiting our ability to measure agents' capabilities in planning, long-context understanding, and memory use. We introduce SWE-Marathon, a benchmark of 20 long-horizon tasks spanning software engineering and adjacent technical domains. Each task consists of a unique executable environment, a human-written reference solution, and a multi-layer verification suite. Logged agent attempts average 27.2M total tokens, making SWE-Marathon substantially longer-horizon than existing SWE and command-line agent benchmarks. Current frontier coding agents solve fewer than 30% of tasks. Failures often arise from poor self-verification, self-reported infeasibility, and premature termination. We also observe reward-hacking behavior in 13.8% of rollouts, where agents attempt to exploit the environment or verifier to bypass the intended workflow. SWE-Marathon includes adversarial review of test suites and execution environments, as well as multi-layer checks designed to prevent shortcut solutions. We release SWE-Marathon, evaluation code, and agent trajectories at https://swe-marathon.org/.