# Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

arXiv: https://arxiv.org/abs/2603.00468 · HTML: https://mlsanity.com/paper/2603.00468/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 71 references that are in the map.
- First posted on arXiv 28 Feb 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [Automatic Root Cause Analysis via Large Language Models for Cloud Incidents](https://mlsanity.com/paper/2305.15778/) (2023)
- [Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics](https://mlsanity.com/paper/2008.06448/) (2020)
- [ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks](https://mlsanity.com/paper/2502.05352/) (2025)
- [AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds](https://mlsanity.com/paper/2501.06706/) (2025)
- [RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data](https://mlsanity.com/paper/2412.17015/) (2024)
- [TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Models](https://mlsanity.com/paper/2402.10802/) (2024)
- [Actionable and Interpretable Fault Localization for Recurring Failures in Online Service Systems](https://mlsanity.com/paper/2207.09021/) (2022)
- [X-lifecycle Learning for Cloud Incident Management using LLMs](https://mlsanity.com/paper/2404.03662/) (2024)
- [COCA: Generative Root Cause Analysis for Distributed Systems with Code Knowledge](https://mlsanity.com/paper/2503.23051/) (2025)
- [MicroRemed: Benchmarking LLMs in Microservices Remediation](https://mlsanity.com/paper/2511.01166/) (2025)
- [DBAIOps: A Reasoning LLM-Enhanced Database Operation and Maintenance System using Knowledge Graphs](https://mlsanity.com/paper/2508.01136/) (2025)
- [AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production](https://mlsanity.com/paper/2509.14647/) (2025)
- [StepFly: Agentic Troubleshooting Guide Automation for Incident Diagnosis](https://mlsanity.com/paper/2510.10074/) (2025)
- [FaaSRCA: Full Lifecycle Root Cause Analysis for Serverless Applications](https://mlsanity.com/paper/2412.02239/) (2024)
- [Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation](https://mlsanity.com/paper/2505.16590/) (2025)
- [AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems](https://mlsanity.com/paper/2601.09393/) (2026)
- [AlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systems](https://mlsanity.com/paper/2601.14912/) (2026)

## Built on by (most cited first, 15 of 15)

- [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026)
- [SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios](https://mlsanity.com/paper/2605.07161/) (2026)
- [AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection](https://mlsanity.com/paper/2608.06790/) (2026)
- [OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories](https://mlsanity.com/paper/2605.29253/) (2026)
- [ORCA-bench: How Ready Are Language Model Agents for Oncall?](https://mlsanity.com/paper/2607.28545/) (2026)
- [CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend](https://mlsanity.com/paper/2604.23455/) (2026)
- [GraphMind: From Operational Traces to Self-Evolving Workflow Automation](https://mlsanity.com/paper/2605.17617/) (2026)
- [What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks](https://mlsanity.com/paper/2606.29159/) (2026)
- [Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations](https://mlsanity.com/paper/2609.30345/) (2026)
- [Evaluating Agentic Code Repair Capabilities in Distributed Systems](https://mlsanity.com/paper/2608.14863/) (2026)
- [Agent-Native Telemetry: Verifiable State-Delta Evidence for Autonomous Operations](https://mlsanity.com/paper/2608.16178/) (2026)
- [SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents](https://mlsanity.com/paper/2609.33023/) (2026)
- [Incident-Arena: Getting agents to the last nine of reliability](https://mlsanity.com/paper/2610.00648/) (2026)
- [Agentic RCA for Internet-Scale Services Using Constrained Creativity](https://mlsanity.com/paper/2610.08622/) (2026)
- [SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data](https://mlsanity.com/paper/2610.16412/) (2026)

## Abstract (arXiv)

LLM agents are increasingly explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose realistic tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Across ten LLM agents, the strongest Joint RCA Accuracy (JRA) reaches 0.76 on OnlineBoutique and 0.68 on TrainTicket, while the corresponding Evidence Closure Rates (ECR) are only 0.38 and 0.15. This outcome--process gap shows that final-answer correctness alone substantially overestimates agents' ability to perform evidence-grounded diagnosis.