# Ranking with Confidence for Large Scale Comparison Data

arXiv: https://arxiv.org/abs/2202.01670 · HTML: https://mlsanity.com/paper/2202.01670/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 3 Feb 2022.

## Builds on (most cited first, 15 of 15)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Simple, Robust and Optimal Ranking from Pairwise Comparisons](https://mlsanity.com/paper/1512.08949/) (2015)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [Spectral MLE: Top-K Rank Aggregation from Pairwise Comparisons](https://mlsanity.com/paper/1504.07218/) (2015)
- [Minimax Rates and Efficient Algorithms for Noisy Sorting](https://mlsanity.com/paper/1710.10388/) (2017)
- [A Nearly Instance Optimal Algorithm for Top-k Ranking under the Multinomial Logit Model](https://mlsanity.com/paper/1707.08238/) (2017)
- [Preference Completion: Large-scale Collaborative Ranking from Pairwise Comparisons](https://mlsanity.com/paper/1507.04457/) (2015)
- [An application of incomplete pairwise comparison matrices for ranking top tennis players](https://mlsanity.com/paper/1611.00538/) (2016)
- [Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation](https://mlsanity.com/paper/1810.08851/) (2018)
- [Ranking and synchronization from pairwise measurements via SVD](https://mlsanity.com/paper/1906.02746/) (2019)
- [Aggregating Incomplete and Noisy Rankings](https://mlsanity.com/paper/2011.00810/) (2020)
- [Scalable Bayesian Preference Learning for Crowds](https://mlsanity.com/paper/1912.01987/) (2019)
- [Adversarial Top-K Ranking](https://mlsanity.com/paper/1602.04567/) (2016)
- [HodgeRank with Information Maximization for Crowdsourced Pairwise Ranking Aggregation](https://mlsanity.com/paper/1711.05957/) (2017)

## Built on by (most cited first, 3 of 3)

- [Confident Feature Ranking](https://mlsanity.com/paper/2307.15361/) (2023)
- [Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation](https://mlsanity.com/paper/2606.08679/) (2026)
- [Quantifying Ranking Uncertainty in LLM Benchmarks](https://mlsanity.com/paper/2607.16259/) (2026)

## Abstract (arXiv)

In this work, we leverage a generative data model considering comparison noise to develop a fast, precise, and informative ranking algorithm from pairwise comparisons that produces a measure of confidence on each comparison. The problem of ranking a large number of items from noisy and sparse pairwise comparison data arises in diverse applications, like ranking players in online games, document retrieval or ranking human perceptions. Although different algorithms are available, we need fast, large-scale algorithms whose accuracy degrades gracefully when the number of comparisons is too small. Fitting our proposed model entails solving a non-convex optimization problem, which we tightly approximate by a sum of quasi-convex functions and a regularization term. Resorting to an iterative reweighted minimization and the Primal-Dual Hybrid Gradient method, we obtain PD-Rank, achieving a Kendall tau 0.1 higher than all comparing methods, even for 10% of wrong comparisons in simulated data matching our data model, and leading in accuracy if data is generated according to the Bradley-Terry model, in both cases faster by one order of magnitude, in seconds. In real data, PD-Rank requires less computational time to achieve the same Kendall tau than active learning methods.