mlsanityData to 14 Oct 2026

SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents

arXiv 2609.33023 ↗First posted 26 Sep 2026cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Benchmarks for site reliability engineering (SRE) agents are typically episodic: one fault is injected, the agent receives an incident task, and its response is scored. Production operation is not. Incidents surface through noisy alerts, overlap in time, and often originate from code or configuration changes. We present SRE-Marathon, a benchmark for long-horizon, continuous SRE operation. An agent is invoked at a fixed cadence with cumulative alert history and a persistent workspace while operating a live two-zone Kubernetes deployment as a fault orchestrator injects overlapping faults according to a seeded, production-calibrated schedule. Curated code and configuration changes deployed through the same build pipeline available for repair. Each run is recorded into a sealed bundle and scored offline: Marathon-Score credits each injected fault for ordered progress through correlation, localization, and repair, with all metrics computed deterministically from recorded system evidence. Across three applications and about sixty faults per run, the best of 10 methods reaches only 41.3 out of 100. Agents often correlate and localize faults, but almost never complete repairs while the faults remain active.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

33 of 54 references.

References · 33

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  5. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  6. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  7. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  8. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  9. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  10. 2023Automatic Root Cause Analysis via Large Language Models for Cloud Incidentscited by 84
  11. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
  12. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Modelscited by 65
21 more
  1. 2024Exploring LLM-based Agents for Root Cause Analysiscited by 61
  2. 2025ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Taskscited by 51
  3. 2025AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Cloudscited by 40
  4. 2025Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysiscited by 34
  5. 2024RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Datacited by 29
  6. 2023Xpert: Empowering Incident Management with Query Recommendations via Large Language Modelscited by 22
  7. 2025STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Cloudscited by 19
  8. 2025A Survey of AIOps in the Era of Large Language Modelscited by 19
  9. 2026Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systemscited by 15
  10. 2024Nissist: An Incident Mitigation Copilot based on Troubleshooting Guidescited by 12
  11. 2025COCA: Generative Root Cause Analysis for Distributed Systems with Code Knowledgecited by 11
  12. 2026SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarioscited by 10
  13. 2025OpsAgent: An Evolving Multi-agent System for Incident Management in Microservicescited by 6
  14. 2025StepFly: Agentic Troubleshooting Guide Automation for Incident Diagnosiscited by 5
  15. 2026OpenRCA 2.0: From Outcome Labels to Causal Process Supervisioncited by 5
  16. 2024Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approachcited by 4
  17. 2026ORCA-bench: How Ready Are Language Model Agents for Oncall?cited by 4
  18. 2026Agentic Observability: Automated Alert Triage for Adobe E-Commercecited by 2
  19. 2025LogPilot: Intent-aware and Scalable Alert Diagnosis for Large-scale Online Service Systemscited by 1
  20. 2026OpsMem: Dual-Memory Reasoning with Cross-Memory Resonance for Failure Diagnosiscited by 1
  21. 2026GALA: Graph-Augmented LLM Agents for Root Cause Analysis and Incident Response in Microservicescited by 1

Built on by

Not cited yet.