mlsanityData to 14 Oct 2026

Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

arXiv 2610.19095 ↗First posted 13 Oct 2026cs.CL cs.LGcited by 0

New in LLM alignment / RLHF / preference optimisation; no direction yet. LLM alignment / RLHF / preference optimisation

Abstract (arXiv)

Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments, prompt subsets and topics, and timing checks. How alike two models' answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established. The leaderboard order barely moves. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a prompt the shift is smaller and clear for two of four judges.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 47 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

24 of 47 references.

References · 24

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  10. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  11. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  12. 2025The Leaderboard Illusioncited by 66
12 more
  1. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 61
  2. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 27
  3. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 26
  4. 2019Discovering Context Effects from Raw Choice Datacited by 14
  5. 2024A Statistical Framework for Ranking LLM-Based Chatbotscited by 12
  6. 2016Pairwise Choice Markov Chainscited by 10
  7. 2020Learning Interpretable Feature Context Effects in Discrete Choicecited by 10
  8. 2020Preference Modeling with Context-Dependent Salient Featurescited by 8
  9. 2020Fundamental Limits of Testing the Independence of Irrelevant Alternatives in Discrete Choicecited by 4
  10. 2025Learning Correlated Reward Models: Statistical Barriers and Opportunitiescited by 3
  11. 2026Pluralistic Leaderboardscited by 2
  12. 2026Attention Limited Reward Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving Your Model Ranking on Chatbot Arena by Vote Rigging
  2. 2026Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
  3. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
  4. 2024Reward Learning From Preference With Ties
  5. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives
  6. 2026Bradley-Terry model under general comparison graphs
  7. 2024Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
  8. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation