# Similar Models, Broken Odds: Where Luce's Axiom Fails in Chatbot Arena

arXiv: https://arxiv.org/abs/2610.19095 · HTML: https://mlsanity.com/paper/2610.19095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation

## Builds on (most cited first, 24 of 24)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [A Minimaximalist Approach to Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2401.04056/) (2024)
- [Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF](https://mlsanity.com/paper/2312.08358/) (2023)
- [β-DPO: Direct Preference Optimization with Dynamic β](https://mlsanity.com/paper/2407.08639/) (2024)
- [The Leaderboard Illusion](https://mlsanity.com/paper/2504.20879/) (2025)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment](https://mlsanity.com/paper/2410.02197/) (2024)
- [Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback](https://mlsanity.com/paper/2410.19133/) (2024)
- [Discovering Context Effects from Raw Choice Data](https://mlsanity.com/paper/1902.03266/) (2019)
- [A Statistical Framework for Ranking LLM-Based Chatbots](https://mlsanity.com/paper/2412.18407/) (2024)
- [Pairwise Choice Markov Chains](https://mlsanity.com/paper/1603.02740/) (2016)
- [Learning Interpretable Feature Context Effects in Discrete Choice](https://mlsanity.com/paper/2009.03417/) (2020)
- [Preference Modeling with Context-Dependent Salient Features](https://mlsanity.com/paper/2002.09615/) (2020)
- [Fundamental Limits of Testing the Independence of Irrelevant Alternatives in Discrete Choice](https://mlsanity.com/paper/2001.07042/) (2020)
- [Learning Correlated Reward Models: Statistical Barriers and Opportunities](https://mlsanity.com/paper/2510.15839/) (2025)
- [Pluralistic Leaderboards](https://mlsanity.com/paper/2606.02547/) (2026)
- [Attention Limited Reward Learning](https://mlsanity.com/paper/2607.04590/) (2026)

## Abstract (arXiv)

Chatbot Arena ranks language models with the Bradley-Terry model, which gives each model one score and so assumes Luce's choice axiom. Choice theory predicts where this fails: near-copies are told apart more decisively than their scores imply. On 1.8 million Arena votes, votes between versions of one model are 1.37 times as decisive as Bradley-Terry predicts, between sizes of one generation 1.26 times and between models of one lab 1.12 times; a release a year later gives consistent multipliers, and the effect survives alternative tie treatments, prompt subsets and topics, and timing checks. How alike two models' answers are worded and formatted tracks the effect. A correction with four to six parameters predicts held-out votes better than Bradley-Terry and than models with hundreds of parameters, mostly by modelling ties; frozen and carried to new releases in a test fixed in advance, its gains are concentrated in ties, and better head-to-head odds are not established. The leaderboard order barely moves. With language-model judges and three-answer menus, a near-copy takes votes mostly from its twin when choices are pooled over prompts (log-odds shift +0.10, 95% interval 0.05 to 0.16); within a prompt the shift is smaller and clear for two of four judges.