mlsanityData to 13 Oct 2026

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

arXiv 2311.17295 ↗First posted 29 Nov 2023cs.CL cs.AIcited by 59

Not on the map. Why?

Abstract (arXiv)

In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons. However, while popular, the system's suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity. We conduct extensive evaluation of Elo behaviour, illustrating that individual Elo computations exhibit volatility and delving into the impact of varying the Elo rating system's hyperparameters. We show that these axioms are not always satisfied raising questions about the reliability of current comparative evaluations of LLMs. If the current use of Elo scores is intended to substitute the costly head-to-head comparison of LLMs, it is crucial to ensure the ranking is as robust as possible. Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting a need for reassessment of existing comparative approaches.

Key facts. Cited by 59 papers in the map (data through 13 Oct 2026). Builds on 26 of its 60 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

26 of 60 references.

References · 26

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  7. 2017Deep Reinforcement Learning that Matterscited by 734
  8. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  9. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  10. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  11. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  12. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
14 more
  1. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  2. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  3. 2020We Need to Talk About Random Splitscited by 70
  4. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 58
  5. 2021Nondeterminism and Instability in Neural Network Optimizationcited by 48
  6. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  7. 2017Knowledge Base Completion: Baselines Strike Backcited by 39
  8. 2019On Model Stability as a Function of Random Seedcited by 31
  9. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23
  10. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  11. 2020Smooth activations and reproducibility in deep networkscited by 15
  12. 2021Synthesizing Irreproducibility in Deep Networkscited by 9
  13. 2021Evaluating Team Skill Aggregation in Online Competitive Gamescited by 4
  14. 2021Elo Ratings for Large Tournaments of Software Agents in Asymmetric Gamescited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 21; 2025: 20; 2026: 18.

59 papers.

Sort:

By area:

  1. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 143
  2. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 101
  3. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  4. 2024Evaluating Language Model Agency through Negotiationscited by 46
  5. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 42
  6. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 41
  7. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  8. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  9. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  10. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  11. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Datacited by 12
  12. 2025An LLM-Based Approach for Insight Generation in Data Analysiscited by 12
38 more
  1. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  2. 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validitycited by 11
  3. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  4. 2025SafeLawBench: Towards Safe Alignment of Large Language Modelscited by 9
  5. 2024SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMscited by 8
  6. 2025am-ELO: A Stable Framework for Arena-based LLM Evaluationcited by 8
  7. 2024Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combatcited by 7
  8. 2025Evaluation of Large Language Models via Coupled Token Generationcited by 7
  9. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 6
  10. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  11. 2025ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  12. 2025Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratingscited by 5
  13. 2025PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzlescited by 5
  14. 2024Compare without Despair: Reliable Preference Evaluation with Generation Separabilitycited by 4
  15. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  16. 2026DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttalcited by 4
  17. 2024Soft Condorcet Optimization for Ranking of General Agentscited by 3
  18. 2024Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Contextcited by 3
  19. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantificationcited by 3
  20. 2026Position: Evaluation Scores Are Perishable Knowledge Claimscited by 3
  21. 2024CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarizationcited by 2
  22. 2025EigenBench: A Comparative Behavioral Measure of Value Alignmentcited by 2
  23. 2025Paragraph Segmentation Revisited: Towards a Standard Task for Structuring Speechcited by 2
  24. 2026Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboardscited by 2
  25. 2026A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulationcited by 2
  26. 2024PRISM: A Design Framework for Open-Source Foundation Model Safetycited by 1
  27. 2024Soft Measures for Extracting Causal Collective Intelligencecited by 1
  28. 2024Project MPG: towards a generalized performance benchmark for LLM capabilitiescited by 1
  29. 2024UIP2P: Unsupervised Instruction-based Image Editing via Edit Reversibility Constraintcited by 1
  30. 2025MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMscited by 1
  31. 2025LEHA-CVQAD: Dataset To Enable Generalized Video Quality Assessment of Compression Artifactscited by 1
  32. 2025LLMs Behind the Scenes: Enabling Narrative Scene Illustrationcited by 1
  33. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  34. 2026MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferencescited by 1
  35. 2026Conspiracy Frame: a Semiotically-Driven Approach for Conspiracy Theories Detectioncited by 1
  36. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1
  37. 2026(Towards) Scalable Reliable Automated Evaluation with Large Language Modelscited by 1
  38. 2025REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resourcescited by 0

50 most cited of 59 All 59 (JSON)