mlsanityData to 7 Oct 2026

On the Limitations of Elo: Real-World Games, are Transitive, not Additive

arXiv 2206.12301 ↗First posted 21 Jun 2022cs.GT cs.LG stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

Real-world competitive games, such as chess, go, or StarCraft II, rely on Elo models to measure the strength of their players. Since these games are not fully transitive, using Elo implicitly assumes they have a strong transitive component that can correctly be identified and extracted. In this study, we investigate the challenge of identifying the strength of the transitive component in games. First, we show that Elo models can fail to extract this transitive component, even in elementary transitive games. Then, based on this observation, we propose an extension of the Elo score: we end up with a disc ranking system that assigns each player two scores, which we refer to as skill and consistency. Finally, we propose an empirical validation on payoff matrices coming from real-world games played by bots and humans.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 16 of its 45 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

16 of 45 references.

References · 16

  1. 2021Convex optimizationcited by 2,479
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  3. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  4. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  5. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  6. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  7. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  8. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  9. 2020Real World Games Look Like Spinning Topscited by 66
  10. 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 50
  11. 2018Re-evaluating Evaluationcited by 47
  12. 2020EigenGame: PCA as a Nash Equilibriumcited by 39
4 more
  1. 2021Measuring the Non-Transitivity in Chesscited by 10
  2. 2019Competing Against Equilibria in Zero-Sum Games with Evolving Payoffscited by 4
  3. 2021Estimating α-Rank by Maximizing Information Gaincited by 4
  4. 2022Learning to Identify Top Elo Ratings: A Dueling Bandits Approachcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 9; 2026: 1.

23 papers.

Sort:

By area:

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2023Nash Learning from Human Feedbackcited by 220
  3. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  4. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
  5. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  6. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  7. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  8. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  9. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  10. 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validitycited by 11
  11. 2023Evaluating Agents using Social Choice Theorycited by 10
  12. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 9
11 more
  1. 2025Re-evaluating Open-ended Evaluation of Large Language Modelscited by 8
  2. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 7
  3. 2024ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictionscited by 4
  4. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignmentcited by 4
  5. 2024Soft Condorcet Optimization for Ranking of General Agentscited by 3
  6. 2023Ordinal Potential-based Player Ratingcited by 2
  7. 2025Is Elo Rating Reliable? A Study Under Model Misspecificationcited by 2
  8. 2025Strength Estimation and Human-Like Strength Adjustment in Gamescited by 1
  9. 2024Online Self-Preferring Language Modelscited by 0
  10. 2025Inclusion Arena: An Open Platform for Evaluating Large Foundation Models with Real-World Appscited by 0
  11. 2026Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignmentcited by 0