# RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data

arXiv: https://arxiv.org/abs/2412.17015 · HTML: https://mlsanity.com/paper/2412.17015/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2024.

## Built on by (most cited first, 29 of 29)

- [Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems](https://mlsanity.com/paper/2603.00468/) (2026)
- [Why Do AI Agents Systematically Fail at Cloud Root Cause Analysis?](https://mlsanity.com/paper/2602.09937/) (2026)
- [GALA: Can Graph-Augmented Large Language Model Agentic Workflows Elevate Root Cause Analysis?](https://mlsanity.com/paper/2508.12472/) (2025)
- [OpenRCA 2.0: From Outcome Labels to Causal Process Supervision](https://mlsanity.com/paper/2606.27154/) (2026)
- [Graph-Free Root Cause Analysis](https://mlsanity.com/paper/2601.21359/) (2026)
- [ORCA-bench: How Ready Are Language Model Agents for Oncall?](https://mlsanity.com/paper/2607.28545/) (2026)
- [TN-AutoRCA: Benchmark Construction and Agentic Framework for Self-Improving Alarm-Based Root Cause Analysis in Telecommunication Networks](https://mlsanity.com/paper/2507.18190/) (2025)
- [OpenDerisk: An Industrial Framework for AI-Driven SRE, with Design, Implementation, and Case Studies](https://mlsanity.com/paper/2510.13561/) (2025)
- [Bridging Temporal and Textual Modalities: A Multimodal Framework for Automated Cloud Failure Root Cause Analysis](https://mlsanity.com/paper/2601.04709/) (2026)
- [CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend](https://mlsanity.com/paper/2604.23455/) (2026)
- [EventADL: Open-Box Anomaly Detection and Localization Framework for Events in Cloud-Based Service Systems](https://mlsanity.com/paper/2605.00936/) (2026)
- [TopoEvo: A Topology-Aware Self-Evolving Multi-Agent Framework for Root Cause Analysis in Microservices](https://mlsanity.com/paper/2605.15611/) (2026)
- [What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks](https://mlsanity.com/paper/2606.29159/) (2026)
- [A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis](https://mlsanity.com/paper/2606.29193/) (2026)
- [TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?](https://mlsanity.com/paper/2608.07899/) (2026)
- [Autonomic Microservice Management via Agentic AI and MAPE-K Integration](https://mlsanity.com/paper/2506.22185/) (2025)
- [STLGT: A Scalable Trace-Based Linear Graph Transformer for Tail Latency Prediction in Microservices](https://mlsanity.com/paper/2604.26422/) (2026)
- [OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning](https://mlsanity.com/paper/2605.02906/) (2026)
- [StableRCA: Robust Graph-Agnostic Mechanism-Level Root Cause Analysis](https://mlsanity.com/paper/2606.05636/) (2026)
- [How Far Can Root Cause Analysis Go on Real-World Telemetry Data?](https://mlsanity.com/paper/2607.13548/) (2026)
- [From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis](https://mlsanity.com/paper/2608.25661/) (2026)
- [Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring](https://mlsanity.com/paper/2609.27069/) (2026)
- [Stochastic Semantic Evidence Graphs: Uncertainty Propagation and Governance for Agentic AI](https://mlsanity.com/paper/2609.29703/) (2026)
- [SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability Agents](https://mlsanity.com/paper/2609.33023/) (2026)
- [Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition](https://mlsanity.com/paper/2609.36686/) (2026)
- [Beyond Conditional Independence: Root Cause Analysis with Deep Causal Models](https://mlsanity.com/paper/2609.36771/) (2026)
- [Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case](https://mlsanity.com/paper/2610.03190/) (2026)
- [When Log Parsing Removes What Matters: Detecting Failures in Distributed GPU Workloads](https://mlsanity.com/paper/2610.14009/) (2026)
- [SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data](https://mlsanity.com/paper/2610.16412/) (2026)

## Abstract (arXiv)

Root cause analysis (RCA) for microservice systems has gained significant attention in recent years. However, there is still no standard benchmark that includes large-scale datasets and supports comprehensive evaluation environments. In this paper, we introduce RCAEval, an open-source benchmark that provides datasets and an evaluation environment for RCA in microservice systems. First, we introduce three comprehensive datasets comprising 735 failure cases collected from three microservice systems, covering various fault types observed in real-world failures. Second, we present a comprehensive evaluation framework that includes fifteen reproducible baselines covering a wide range of RCA approaches, with the ability to evaluate both coarse-grained and fine-grained RCA. We hope that this ready-to-use benchmark will enable researchers and practitioners to conduct extensive analysis and pave the way for robust new solutions for RCA of microservice systems.