# Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores

arXiv: https://arxiv.org/abs/2509.23510 · HTML: https://mlsanity.com/paper/2509.23510/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement](https://mlsanity.com/paper/2402.11436/) (2024)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [The Diminishing Returns of Masked Language Models to Science](https://mlsanity.com/paper/2205.11342/) (2022)
- [Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference](https://mlsanity.com/paper/2501.00560/) (2024)
- [Unsupervised Elicitation of Language Models](https://mlsanity.com/paper/2506.10139/) (2025)
- [TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences](https://mlsanity.com/paper/2412.00539/) (2024)

## Built on by (most cited first, 1 of 1)

- [PACE: A Proxy for Agentic Capability Evaluation](https://mlsanity.com/paper/2607.02032/) (2026)

## Abstract (arXiv)

New large language models (LLMs) are being released every day. Some perform significantly better or worse than expected given their parameter count. Therefore, there is a need for a method to independently evaluate models. The current best way to evaluate a model is to measure its Elo score by comparing it to other models in a series of contests - an expensive operation since humans are ideally required to compare LLM outputs. We observe that when an LLM is asked to judge such contests, the consistency with which it selects a model as the best in a matchup produces a metric that is 91% correlated with its own human-produced Elo score. This provides a simple proxy for Elo scores that can be computed cheaply, without any human data or prior knowledge.