# What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks

arXiv: https://arxiv.org/abs/2606.29159 · HTML: https://mlsanity.com/paper/2606.29159/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Causal Inference-Based Root Cause Analysis for Online Service Systems with Intervention Recognition](https://mlsanity.com/paper/2206.05871/) (2022)
- [ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks](https://mlsanity.com/paper/2502.05352/) (2025)
- [AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds](https://mlsanity.com/paper/2501.06706/) (2025)
- [RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data](https://mlsanity.com/paper/2412.17015/) (2024)
- [BARO: Robust Root Cause Analysis for Microservices via Multivariate Bayesian Online Change Point Detection](https://mlsanity.com/paper/2405.09330/) (2024)
- [Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?](https://mlsanity.com/paper/2408.13729/) (2024)
- [Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems](https://mlsanity.com/paper/2603.00468/) (2026)
- [The PetShop Dataset -- Finding Causes of Performance Issues across Microservices](https://mlsanity.com/paper/2311.04806/) (2023)
- [LEMMA-RCA: A Large Multi-modal Multi-domain Dataset for Root Cause Analysis](https://mlsanity.com/paper/2406.05375/) (2024)
- [Why Do AI Agents Systematically Fail at Cloud Root Cause Analysis?](https://mlsanity.com/paper/2602.09937/) (2026)
- [Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis](https://mlsanity.com/paper/2601.22208/) (2026)
- [Rethinking the Evaluation of Microservice RCA with a Fault Propagation-Aware Benchmark](https://mlsanity.com/paper/2510.04711/) (2025)

## Built on by (most cited first, 1 of 1)

- [Incident-Arena: Getting agents to the last nine of reliability](https://mlsanity.com/paper/2610.00648/) (2026)

## Abstract (arXiv)

Offline benchmarks for microservice root cause analysis (RCA) rank methods by a score pooled over several applications, and a pooled lead leaves two things unstated: on which of the systems it holds, and how the score was produced. We audit both on three public benchmark families (OpenRCA, RCAEval, and PetShop), running the released implementations of 16 methods under recorded configurations on 11 subsystems and on a second release of three of the applications. Whether a pooled lead between published methods holds on every subsystem is decided by a quantity the leaderboard withholds, the spread of the lead across subsystems: of 15 leads whose margin exceeds this spread, 14 hold everywhere; of the 8 whose margin is below it, 6 reverse. The per-fault and per-scenario tables that the benchmark papers publish split the same way, and every error of a ranking computed without one subsystem falls on a lead of the second kind. On the second release a third of the comparisons change sign, most through one method's response to the metric kinds it records, and on several subsystems a simple probe or a shipped baseline beats every published method. How the score was produced matters as much: the input table that one released runner reads is not the one its documentation describes, and this choice alone reverses a comparison between published methods with intervals excluding zero both ways; an unpinned dependency and a duplicated column header each make a method return its input column order without an error. Of 23 RCA papers we examined, none releases the per-case outputs needed to check either premise. We recommend that a pooled comparison state the subsystems it covers and its spread across them, the input table, the handling of failed outputs, and the denominator, and be accompanied by per-case outputs, as ours are in our replication package.