# am-ELO: A Stable Framework for Arena-based LLM Evaluation

arXiv: https://arxiv.org/abs/2505.03475 · HTML: https://mlsanity.com/paper/2505.03475/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 59 references that are in the map.
- First posted on arXiv 6 May 2025.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Generative Judge for Evaluating Alignment](https://mlsanity.com/paper/2310.05470/) (2023)
- [EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction](https://mlsanity.com/paper/1906.05658/) (2019)
- [AgentReview: Exploring Peer Review Dynamics with LLM Agents](https://mlsanity.com/paper/2406.12708/) (2024)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms](https://mlsanity.com/paper/2402.14154/) (2024)
- [LLMEval: A Preliminary Study on How to Evaluate Large Language Models](https://mlsanity.com/paper/2312.07398/) (2023)
- [Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform](https://mlsanity.com/paper/2403.08305/) (2024)
- [Evaluatology: The Science and Engineering of Evaluation](https://mlsanity.com/paper/2404.00021/) (2024)
- [A Spectral Approach to Item Response Theory](https://mlsanity.com/paper/2210.04317/) (2022)
- [An Elo-like System for Massive Multiplayer Competitions](https://mlsanity.com/paper/2101.00400/) (2021)

## Built on by (most cited first, 8 of 8)

- [PointArena: Probing Multimodal Grounding Through Language-Guided Pointing](https://mlsanity.com/paper/2505.09990/) (2025)
- [PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature](https://mlsanity.com/paper/2510.10909/) (2025)
- [Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?](https://mlsanity.com/paper/2506.04575/) (2025)
- [Toward a unified framework for data-efficient evaluation of large language models](https://mlsanity.com/paper/2510.04051/) (2025)
- [EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents](https://mlsanity.com/paper/2604.05557/) (2026)
- [From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation](https://mlsanity.com/paper/2606.13221/) (2026)
- [Robust AI Evaluation through Maximal Lotteries](https://mlsanity.com/paper/2602.21297/) (2026)
- [The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations](https://mlsanity.com/paper/2507.13302/) (2025)

## Abstract (arXiv)

Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating's probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs.