mlsanityData to 14 Oct 2026

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

arXiv 2508.11847 ↗First posted 16 Aug 2025stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM ranking platforms, including Chatbot Arena and derivatives, we find that the rankings of top-performing models can be remarkably sensitive to the removal of a small fraction of preferences; for instance, dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena. Our robustness check identifies the specific preferences most responsible for such ranking flips, allowing for inspection of these influential preferences. We observe that the rankings derived from MT-bench preferences are notably more robust than those from Chatbot Arena, likely due to MT-bench's use of expert annotators and carefully constructed prompts. Finally, we find that neither rankings based on crowdsourced human evaluations nor those based on LLM-as-a-judge preferences are systematically more sensitive than the other.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 16 Aug 2025.

Builds on

37 of 59 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  10. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 371
  11. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  12. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
25 more
  1. 2023Efficient Benchmarking of Language Modelscited by 71
  2. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 70
  3. 2025The Leaderboard Illusioncited by 66
  4. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62
  5. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  6. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 44
  7. 2022Algorithms that Approximate Data Removal: New Results and Limitationscited by 40
  8. 2025Search Arena: Analyzing Search-Augmented LLMscited by 34
  9. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  10. 2021Uncertainty quantification in the Bradley-Terry-Luce modelcited by 29
  11. 2020An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?cited by 27
  12. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  13. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 20
  14. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 17
  15. 2019A Higher-Order Swiss Army Infinitesimal Jackknifecited by 14
  16. 2020Approximate Cross-validation: Guarantees for Model Assessment and Selectioncited by 14
  17. 2024Challenges in Trustworthy Human Evaluation of Chatbotscited by 14
  18. 2024The Future of Open Human Feedbackcited by 10
  19. 2020Approximate Cross-Validation for Structured Modelscited by 9
  20. 2024Approximations to worst-case data dropping: unmasking failure modescited by 8
  21. 2022Asymptotic comparison of identifying constraints for Bradley-Terry modelscited by 6
  22. 2022Provably Auditing Ordinary Least Squares in Low Dimensionscited by 6
  23. 2023Towards Practical Robustness Auditing for Linear Regressioncited by 6
  24. 2024Robustness Auditing for Linear Regression: To Singularity and Beyondcited by 6
  25. 2023Could dropping a few cells change the takeaways from differential expression?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 10
  2. 2026A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulationcited by 2
  3. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 2
  4. 2026Strategic Candidacy in Generative AI Arenascited by 1
  5. 2026Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processescited by 1
  6. 2026RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficultycited by 0
  7. 2026Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised MLcited by 0
  8. 2026The Trust Paradox: How CS Researchers Engage LLM Leaderboardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Personalized Benchmarking: Evaluating LLMs by Individual Preferences
  2. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
  3. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
  4. 2026Nonparametric LLM Evaluation from Preference Data
  5. 2024CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models
  6. 2026Robust AI Evaluation through Maximal Lotteries
  7. 2026Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
  8. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger