# Rank Confidence Sequences:Anytime-valid Leaderboards

arXiv: https://arxiv.org/abs/2609.32211 · HTML: https://mlsanity.com/paper/2609.32211/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 30 of 31)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Time-uniform, nonparametric, nonasymptotic confidence sequences](https://mlsanity.com/paper/1810.08240/) (2018)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Black-Box Reductions for Parameter-free Online Learning in Banach Spaces](https://mlsanity.com/paper/1802.06293/) (2018)
- [Parallelizing the dual revised simplex method](https://mlsanity.com/paper/1503.01889/) (2015)
- [The Leaderboard Illusion](https://mlsanity.com/paper/2504.20879/) (2025)
- [Confidence sequences for sampling without replacement](https://mlsanity.com/paper/2006.04347/) (2020)
- [Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints](https://mlsanity.com/paper/2503.01747/) (2025)
- [Resolution Diagnostics for Paired LLM Evaluation](https://mlsanity.com/paper/2605.30315/) (2026)
- [Confident Feature Ranking](https://mlsanity.com/paper/2307.15361/) (2023)
- [Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation](https://mlsanity.com/paper/2606.08679/) (2026)
- [An improvement of Tukey's HSD with application to ranking institutions](https://mlsanity.com/paper/1708.02428/) (2017)
- [CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes](https://mlsanity.com/paper/2606.20820/) (2026)
- [Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification](https://mlsanity.com/paper/2603.03336/) (2026)
- [Efficient Sequential Evaluation of Large Language Models](https://mlsanity.com/paper/2607.17409/) (2026)
- [Sequential model confidence sets](https://mlsanity.com/paper/2404.18678/) (2024)
- [Admissible online closed testing must employ e-values](https://mlsanity.com/paper/2407.15733/) (2024)
- [Finite-Sample Valid Rank Confidence Sets for a Broad Class of Statistical and Machine Learning Models](https://mlsanity.com/paper/2512.00316/) (2025)
- [Family-wise Error Rate Control with E-values](https://mlsanity.com/paper/2501.09015/) (2025)
- [Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons](https://mlsanity.com/paper/2605.29395/) (2026)
- [Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data](https://mlsanity.com/paper/2607.08522/) (2026)
- [Quantifying Ranking Uncertainty in LLM Benchmarks](https://mlsanity.com/paper/2607.16259/) (2026)

## Built on by (most cited first, 2 of 2)

- [Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging](https://mlsanity.com/paper/2610.10597/) (2026)
- [When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark](https://mlsanity.com/paper/2610.16067/) (2026)

## Abstract (arXiv)

Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level α, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.