# Uncertainty in Ranking

arXiv: https://arxiv.org/abs/2107.03459 · HTML: https://mlsanity.com/paper/2107.03459/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2021.

## Built on by (most cited first, 6 of 6)

- [Prediction-Powered Ranking of Large Language Models](https://mlsanity.com/paper/2402.17826/) (2024)
- [Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks](https://mlsanity.com/paper/2501.04234/) (2025)
- [Confident Feature Ranking](https://mlsanity.com/paper/2307.15361/) (2023)
- [Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation](https://mlsanity.com/paper/2606.08679/) (2026)
- [Quantifying Ranking Uncertainty in LLM Benchmarks](https://mlsanity.com/paper/2607.16259/) (2026)
- [Distribution-informed Efficient Conformal Prediction for Full Ranking](https://mlsanity.com/paper/2601.23128/) (2026)

## Abstract (arXiv)

Ranks estimated from data are uncertain and this poses a challenge in many applications. However, estimated ranks are deterministic functions of estimated parameters, so the uncertainty in the ranks must be determined by the uncertainty in the parameter estimates. We give a complete characterization of this relationship in terms of the linear extensions of a partial order determined by interval estimates of the parameters of interest. We then use this relationship to give a set estimator for the overall ranking, use its size to measure the uncertainty in a ranking, and give efficient algorithms for several questions of interest. We show that our set estimator is a valid confidence set and describe its relationship to a joint confidence set for ranks recently proposed by Klein, Wright & Wieczorek. We apply our methods to both simulated and real data and make them available through the R package rankUncertainty.