# How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?

arXiv: https://arxiv.org/abs/2609.28177 · HTML: https://mlsanity.com/paper/2609.28177/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Exact post-selection inference, with application to the lasso](https://mlsanity.com/paper/1311.6238/) (2013)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [The Ladder: A Reliable Leaderboard for Machine Learning Competitions](https://mlsanity.com/paper/1502.04585/) (2015)
- [The Leaderboard Illusion](https://mlsanity.com/paper/2504.20879/) (2025)
- [Correlated Errors in Large Language Models](https://mlsanity.com/paper/2506.07962/) (2025)
- [Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints](https://mlsanity.com/paper/2503.01747/) (2025)
- [Improving Your Model Ranking on Chatbot Arena by Vote Rigging](https://mlsanity.com/paper/2501.17858/) (2025)
- [Rank Verification for Exponential Families](https://mlsanity.com/paper/1610.03944/) (2016)
- [Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking](https://mlsanity.com/paper/2604.11581/) (2026)
- [A Flexible Defense Against the Winner's Curse](https://mlsanity.com/paper/2411.18569/) (2024)
- [Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking](https://mlsanity.com/paper/2605.05973/) (2026)
- [Resolution Diagnostics for Paired LLM Evaluation](https://mlsanity.com/paper/2605.30315/) (2026)
- [Cluster-Robust Variance Estimation for Dyadic Data](https://mlsanity.com/paper/1312.3398/) (2013)
- [Climbing a shaky ladder: Better adaptive risk estimation](https://mlsanity.com/paper/1706.02733/) (2017)
- [Locally Simultaneous Inference](https://mlsanity.com/paper/2212.09009/) (2022)
- [Critical Values Robust to P-hacking](https://mlsanity.com/paper/2005.04141/) (2020)
- [Powerful rank verification for multivariate Gaussian data with any covariance structure](https://mlsanity.com/paper/2503.01065/) (2025)
- [Inference conditional on selection: a review](https://mlsanity.com/paper/2604.09779/) (2026)
- [Flexible Inference for Winners with Conditional Validity](https://mlsanity.com/paper/2607.18545/) (2026)

## Abstract (arXiv)

LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statistical evidence of a provider's advantage over a fixed comparator. For a fixed candidate family under a Gaussian margin model, we derive a sensitivity curve that reports this maximum count as a function of a lower bound on within-family correlation. The relevant correlation must match the score used for ranking and the sampling model: in a controlled family, pooled item correlation is 0.90, whereas composite-score correlation is 0.46 under item resampling and 0.92 when MMLU subjects are resampled. An item-based audit of 394 adjacent-rank claims on the Open LLM Leaderboard finds that 391 lack statistical support even before accounting for selection. Among claims that pass the uncorrected test, certification can depend on assumptions about the hidden family's correlation. The resulting curves make these assumptions explicit without estimating the unobserved search size.