# Quantifying Ranking Uncertainty in LLM Benchmarks

arXiv: https://arxiv.org/abs/2607.16259 · HTML: https://mlsanity.com/paper/2607.16259/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks](https://mlsanity.com/paper/2501.04234/) (2025)
- [Uncertainty in Ranking](https://mlsanity.com/paper/2107.03459/) (2021)
- [Confident Feature Ranking](https://mlsanity.com/paper/2307.15361/) (2023)
- [An improvement of Tukey's HSD with application to ranking institutions](https://mlsanity.com/paper/1708.02428/) (2017)
- [Ranking with Confidence for Large Scale Comparison Data](https://mlsanity.com/paper/2202.01670/) (2022)
- [Finite-Sample Valid Rank Confidence Sets for a Broad Class of Statistical and Machine Learning Models](https://mlsanity.com/paper/2512.00316/) (2025)
- [Statistical multi-metric evaluation and visualization of LLM system predictive performance](https://mlsanity.com/paper/2501.18243/) (2025)

## Built on by (most cited first, 2 of 2)

- [Rank Confidence Sequences:Anytime-valid Leaderboards](https://mlsanity.com/paper/2609.32211/) (2026)
- [Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?](https://mlsanity.com/paper/2609.00482/) (2026)

## Abstract (arXiv)

Pretrained models are typically ranked on multi-task leaderboards to assess their effectiveness across diverse tasks. Rank confidence intervals were recently introduced as a method to quantify the uncertainty in these rankings by aggregating pairwise hypothesis tests. In this work, we analyze the sources of uncertainty in the knowledge evaluation benchmark MMLU and show how hypothesis tests can be modified to account for their effects. We demonstrate that ranking variability across MMLU subjects is substantial and should be considered when comparing LLMs or identifying the top-performing models.