# Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons

arXiv: https://arxiv.org/abs/2411.01281 · HTML: https://mlsanity.com/paper/2411.01281/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 30 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench](https://mlsanity.com/paper/2407.13696/) (2024)

## Built on by (most cited first, 9 of 9)

- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [Stochasticity in Agentic Evaluations: Quantifying Inconsistency with Intraclass Correlation](https://mlsanity.com/paper/2512.06710/) (2025)
- [Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis](https://mlsanity.com/paper/2510.08759/) (2025)
- [Mediocrity is the key for LLM as a Judge Anchor Selection](https://mlsanity.com/paper/2603.16848/) (2026)
- [Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation](https://mlsanity.com/paper/2606.08679/) (2026)
- [Dynamically Allocating Evaluation Effort for Model Ranking](https://mlsanity.com/paper/2608.03437/) (2026)
- [Capturing LLM Capabilities via Evidence-Calibrated Query Clustering](https://mlsanity.com/paper/2605.17110/) (2026)
- [SWE-Duel: Towards a Self-Scaling Arena for Coding Agents](https://mlsanity.com/paper/2610.12569/) (2026)
- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) expand across domains, LLM judges have become essential for systems evaluation. Current benchmarks typically compare system outputs against baselines. This baseline-mediated approach, though convenient, yields lower reliability than direct comparison between systems. We propose Arena-Lite which integrates tournament structure on top of head-to-head comparison. The application of a tournament structure and direct comparison eliminates the need for baseline outputs, reduces the number of required comparisons, and allows higher reliability in system rankings. We conducted two experiments: (1) controlled stochastic modeling and (2) empirical validation with a real LLM judge. Those experiments collectively demonstrate that Arena-Lite consistently achieves higher reliability with fewer comparisons, even with smaller datasets or weaker judges. We release an easy-to-use web demonstration and code to foster adoption of Arena-Lite, streamlining model selection across research and industry communities. Arena-Lite demo and code are available on hrefhttps://huggingface.co/spaces/NCSOFT/ArenaLitehttps://huggingface.co/spaces/NCSOFT/ArenaLite