# SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios arXiv: https://arxiv.org/abs/2605.07161 ยท HTML: https://mlsanity.com/paper/2605.07161/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 23 of its 88 references that are in the map. - First posted on arXiv 8 May 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 23 of 23) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026) - [ADBench: Anomaly Detection Benchmark](https://mlsanity.com/paper/2206.09426/) (2022) - [Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://mlsanity.com/paper/2507.02825/) (2025) - [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024) - [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://mlsanity.com/paper/2305.15778/) (2023) - [RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models](https://mlsanity.com/paper/2310.16340/) (2023) - [Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series](https://mlsanity.com/paper/2010.05073/) (2020) - [ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks](https://mlsanity.com/paper/2502.05352/) (2025) - [AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds](https://mlsanity.com/paper/2501.06706/) (2025) - [Flow-of-Action: SOP Enhanced LLM-Based Multi-Agent System for Root Cause Analysis](https://mlsanity.com/paper/2502.08224/) (2025) - [mABC: multi-Agent Blockchain-Inspired Collaboration for root cause analysis in micro-services architecture](https://mlsanity.com/paper/2404.12135/) (2024) - [STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds](https://mlsanity.com/paper/2506.02009/) (2025) - [Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems](https://mlsanity.com/paper/2603.00468/) (2026) - [TAMO: Fine-Grained Root Cause Analysis via Tool-Assisted LLM Agent with Multi-Modality Observation Data in Cloud-Native Systems](https://mlsanity.com/paper/2504.20462/) (2025) - [Adaptive Root Cause Localization for Microservice Systems with Multi-Agent Recursion-of-Thought](https://mlsanity.com/paper/2508.20370/) (2025) - [Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation](https://mlsanity.com/paper/2409.07355/) (2024) - [MicroRemed: Benchmarking LLMs in Microservices Remediation](https://mlsanity.com/paper/2511.01166/) (2025) - [GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis?](https://mlsanity.com/paper/2508.12472/) (2025) - [StepFly: Agentic Troubleshooting Guide Automation for Incident Diagnosis](https://mlsanity.com/paper/2510.10074/) (2025) - [MicroRCA-Agent: Microservice Root Cause Analysis Method Based on Large Language Model Agents](https://mlsanity.com/paper/2509.15635/) (2025) - [Configuration Defects in Kubernetes](https://mlsanity.com/paper/2512.05062/) (2025) ## Built on by (most cited first, 10 of 10) - [Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions](https://mlsanity.com/paper/2607.02294/) (2026) - [ORCA-bench: How Ready Are Language Model Agents for Oncall?](https://mlsanity.com/paper/2607.28545/) (2026) - [A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis](https://mlsanity.com/paper/2606.29193/) (2026) - [InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk](https://mlsanity.com/paper/2608.11234/) (2026) - [Towards a Systems Foundation for Agentic Cloud Management](https://mlsanity.com/paper/2607.25883/) (2026) - [TriFleetRCA: On-Premise LLM Root Cause Analysis for Kubernetes](https://mlsanity.com/paper/2609.23766/) (2026) - [SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents](https://mlsanity.com/paper/2609.33023/) (2026) - [Incident-Arena: Getting agents to the last nine of reliability](https://mlsanity.com/paper/2610.00648/) (2026) - [Agentic RCA for Internet-Scale Services Using Constrained Creativity](https://mlsanity.com/paper/2610.08622/) (2026) - [A Self-Defining Operator for Distributed Applications](https://mlsanity.com/paper/2610.14547/) (2026) ## Abstract (arXiv) AI agents are increasingly used to diagnose and mitigate failures in production systems, known as agentic Site Reliability Engineering (SRE). Current SRE benchmarks are limited to oversimplistic SRE tasks and are unfortunately hard to extend due to bespoke designs. We present SREGym, a high-fidelity benchmark for SRE agents. SREGym exposes a live system environment built atop real-world cloud-native system stacks, where high-fidelity failure scenarios are simulated through fault injectors. SREGym models the complexity of production environments by simulating (1) a wide range of faults at different layers, (2) various ambient noises, and (3) diverse failure modes such as metastable failures and correlated failures. SREGym is architected as a modular, extensible framework that orchestrates fault and noise injectors across stacks. SREGym currently includes 90 realistic, challenging SRE problems. We use SREGym to evaluate frontier agents and show that their capabilities varies significantly in addressing different kinds of failures, with up to 40% differences in end-to-end results. SREGym is actively maintained as an open-source project and has been used by researchers and practitioners.