mlsanityData to 14 Oct 2026

Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML

arXiv 2605.06656 ↗First posted 7 May 2026cs.LG cs.DM cs.ETcited by 0

Not on the map. Why?

Abstract (arXiv)

Ranking LLMs via pairwise human feedback underpins current leaderboards for open-ended tasks, such as creative writing and problem-solving. We analyze ~89K comparisons in 116 languages from 52 LLMs from Arena, and show that the best-fit global Bradley-Terry (BT) ranking is misleading. Nearly 2/3 of the decisive votes cancel out, and even the top 50 models according to the global BT ranking are statistically indistinguishable (pairwise win probabilities are at most 0.53 within the top 50 models). We trace this failure to strong, structured heterogeneity of opinions across language, task, and time. Moreover, we find an important characteristic - *language* plays a key role. Grouping by language (and families) increases the agreement of votes massively, resulting in two orders of magnitude higher spread in the ELO scores (i.e., very consistent rankings). What appears as global noise is in fact a mixture of coherent but conflicting subpopulations. To address such heterogeneity in supervised machine learning, we introduce the framework of (λ, nu)-portfolios, which are small sets of models that achieve a prediction error at most λ, "covering" at least a nu fraction of users. We formulate this as a variant of the set cover problem and provide guarantees using the VC dimension of the underlying set system. On the Arena data, our algorithms recover just 5 distinct BT rankings that cover over 96% of votes at a modest λ, compared to the 21% coverage by the global ranking. We also provide a portfolio of 6 LLMs that cover twice as many votes as the top-6 LLMs from a global ranking. We further construct portfolios for a classification problem on the COMPAS dataset using an ensemble of fairness-regularized classification models and show that these portfolios can be used to detect blind spots in the data, which might be of independent interest to policymakers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

22 of 44 references.

References · 22

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  3. 2017Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairnesscited by 384
  4. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  5. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  6. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  7. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  8. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  9. 2025The Leaderboard Illusioncited by 66
  10. 2019Metric Learning for Individual Fairnesscited by 52
  11. 2019Average Individual Fairness: Algorithms, Generalization and Experimentscited by 46
  12. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
10 more
  1. 2018Individual Fairness in Hindsightcited by 33
  2. 2020Metric-Free Individual Fairness in Online Learningcited by 30
  3. 2020Individual Fairness in Pipelinescited by 22
  4. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 20
  5. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 17
  6. 2022Which Lp norm is the fairest? Approximations for fair facility location across all "p"cited by 9
  7. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankingscited by 8
  8. 2023FIRE: An Optimization Approach for Fast Interpretable Rule Extractioncited by 5
  9. 2025Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learningcited by 5
  10. 2025Aggregating Concepts of Fairness and Accuracy in Prediction Algorithmscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
  2. 2026Nonparametric LLM Evaluation from Preference Data
  3. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
  4. 2026Personalized Benchmarking: Evaluating LLMs by Individual Preferences
  5. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
  6. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  7. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
  8. 2025User-centric Subjective Leaderboard by Customizable Reward Modeling