mlsanityData to 14 Oct 2026

Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

arXiv 2603.00468 ↗First posted 28 Feb 2026cs.SEcited by 15

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM agents are increasingly explored for automating root cause analysis (RCA) in cloud-native systems, creating a need to evaluate both diagnostic correctness and the quality of the supporting investigation. Existing static benchmarks offer repeatable inputs but limited system-facing interaction, while live testbeds expose realistic tools but hinder controlled comparison because incident evidence varies across runs; both paradigms focus primarily on final answers. To address these limitations, we present Cloud-OpsBench, an evaluation infrastructure for interactive and evidence-grounded cloud RCA. It comprises 754 runtime-verified cases across 57 fault types on two microservice workloads spanning application services and Kubernetes platform layers. Each fault is captured as a state snapshot and replayed through standard diagnostic interfaces, with outcome labels and diagnostic evidence graphs that enable matched comparisons and process-level analysis. Across ten LLM agents, the strongest Joint RCA Accuracy (JRA) reaches 0.76 on OnlineBoutique and 0.68 on TrainTicket, while the corresponding Evidence Closure Rates (ECR) are only 0.38 and 0.15. This outcome--process gap shows that final-answer correctness alone substantially overestimates agents' ability to perform evidence-grounded diagnosis.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 71 references that are in the map. First posted on arXiv 28 Feb 2026.

Builds on

27 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025StepFly: Agentic Troubleshooting Guide Automation for Incident Diagnosis11 Oct
  2. 2025AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production18 Sep
  3. 2025AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds12 Jan

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  9. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  10. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  11. 2023Automatic Root Cause Analysis via Large Language Models for Cloud Incidentscited by 84
  12. 2020Loghub: A Large Collection of System Log Datasets for AI-driven Log Analyticscited by 70
12 more
  1. 2025ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Taskscited by 51
  2. 2024RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Datacited by 29
  3. 2024TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Modelscited by 23
  4. 2022Actionable and Interpretable Fault Localization for Recurring Failures in Online Service Systemscited by 19
  5. 2024X-lifecycle Learning for Cloud Incident Management using LLMscited by 19
  6. 2025COCA: Generative Root Cause Analysis for Distributed Systems with Code Knowledgecited by 11
  7. 2025MicroRemed: Benchmarking LLMs in Microservices Remediationcited by 11
  8. 2025DBAIOps: A Reasoning LLM-Enhanced Database Operation and Maintenance System using Knowledge Graphscited by 7
  9. 2024FaaSRCA: Full Lifecycle Root Cause Analysis for Serverless Applicationscited by 3
  10. 2025Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generationcited by 3
  11. 2026AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systemscited by 3
  12. 2026AlertGuardian: Intelligent Alert Life-Cycle Management for Large-scale Cloud Systemscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 12
  2. 2026SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarioscited by 10
  3. 2026AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injectioncited by 5
  4. 2026OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectoriescited by 4
  5. 2026ORCA-bench: How Ready Are Language Model Agents for Oncall?cited by 4
  6. 2026CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backendcited by 1
  7. 2026GraphMind: From Operational Traces to Self-Evolving Workflow Automationcited by 1
  8. 2026What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarkscited by 1
  9. 2026Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigationscited by 1
  10. 2026Evaluating Agentic Code Repair Capabilities in Distributed Systemscited by 0
  11. 2026Agent-Native Telemetry: Verifiable State-Delta Evidence for Autonomous Operationscited by 0
  12. 2026SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agentscited by 0
3 more
  1. 2026Incident-Arena: Getting agents to the last nine of reliabilitycited by 0
  2. 2026Agentic RCA for Internet-Scale Services Using Constrained Creativitycited by 0
  3. 2026SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis
  2. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
  3. 2026Why Do AI Agents Systematically Fail at Cloud Root Cause Analysis?
  4. 2026Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis
  5. 2026STAR: A Stage-attributed Triage and Repair framework for RCA Agents in Microservices
  6. 2025PRAXIS: Integrating Program Analysis with Observability for Root-Cause Analysis
  7. 2026How Far Can Root Cause Analysis Go on Real-World Telemetry Data?
  8. 2025TAMO: Fine-Grained Root Cause Analysis via Tool-Assisted LLM Agent with Multi-Modality Observation Data in Cloud-Native Systems