# Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

arXiv: https://arxiv.org/abs/2411.14483 · HTML: https://mlsanity.com/paper/2411.14483/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 9 of 9)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [Measuring the Non-Transitivity in Chess](https://mlsanity.com/paper/2110.11737/) (2021)

## Built on by (most cited first, 8 of 8)

- [Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference](https://mlsanity.com/paper/2501.00560/) (2024)
- [Estimating problem difficulty without ground truth using Large Language Model comparisons](https://mlsanity.com/paper/2512.14220/) (2025)
- [A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation](https://mlsanity.com/paper/2605.15761/) (2026)
- [Qomhra: A Bilingual Irish and English Large Language Model](https://mlsanity.com/paper/2510.17652/) (2025)
- [SLMEval: Entropy-Based Calibration for Human-Aligned Evaluation of Large Language Models](https://mlsanity.com/paper/2505.16003/) (2025)
- [The Trust Paradox: How CS Researchers Engage LLM Leaderboards](https://mlsanity.com/paper/2605.28966/) (2026)
- [Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs](https://mlsanity.com/paper/2606.17634/) (2026)
- [Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments](https://mlsanity.com/paper/2610.15634/) (2026)

## Abstract (arXiv)

Deciding which large language model (LLM) to use is a complex challenge. Pairwise ranking has emerged as a new method for evaluating human preferences for LLMs. This approach entails humans evaluating pairs of model outputs based on a predefined criterion. By collecting these comparisons, a ranking can be constructed using methods such as Elo. However, applying these algorithms as constructed in the context of LLM evaluation introduces several challenges. In this paper, we explore the effectiveness of ranking systems for head-to-head comparisons of LLMs. We formally define a set of fundamental principles for effective ranking and conduct a series of extensive evaluations on the robustness of several ranking algorithms in the context of LLMs. Our analysis uncovers key insights into the factors that affect ranking accuracy and efficiency, offering guidelines for selecting the most appropriate methods based on specific evaluation contexts and resource constraints.