# Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

arXiv: https://arxiv.org/abs/2501.00560 · HTML: https://mlsanity.com/paper/2501.00560/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 31 Dec 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://mlsanity.com/paper/2308.01862/) (2023)
- [MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://mlsanity.com/paper/2406.06565/) (2024)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)
- [JuStRank: Benchmarking LLM Judges for System Ranking](https://mlsanity.com/paper/2412.09569/) (2024)
- [ReIFE: Re-evaluating Instruction-Following Evaluation](https://mlsanity.com/paper/2410.07069/) (2024)
- [Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat](https://mlsanity.com/paper/2411.14483/) (2024)

## Built on by (most cited first, 16 of 16)

- [When LLM Judge Scores Look Good but Best-of-N Decisions Fail](https://mlsanity.com/paper/2603.12520/) (2026)
- [Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders](https://mlsanity.com/paper/2507.23220/) (2025)
- [Mediocrity is the key for LLM as a Judge Anchor Selection](https://mlsanity.com/paper/2603.16848/) (2026)
- [Seven simple steps for log analysis in AI systems](https://mlsanity.com/paper/2604.09563/) (2026)
- [REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems](https://mlsanity.com/paper/2608.10669/) (2026)
- [Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation](https://mlsanity.com/paper/2506.05062/) (2025)
- [On Evaluating LLM Alignment by Evaluating LLMs as Judges](https://mlsanity.com/paper/2511.20604/) (2025)
- [Performance of GPT-5 Frontier Models in Ophthalmology Question Answering](https://mlsanity.com/paper/2508.09956/) (2025)
- [Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressions](https://mlsanity.com/paper/2510.08915/) (2025)
- [A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation](https://mlsanity.com/paper/2605.15761/) (2026)
- [Model Consistency as a Cheap yet Predictive Proxy for LLM Elo Scores](https://mlsanity.com/paper/2509.23510/) (2025)
- [HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation](https://mlsanity.com/paper/2604.09629/) (2026)
- [Aligning Language Model Benchmarks with Pairwise Preferences](https://mlsanity.com/paper/2602.02898/) (2026)
- [The Trust Paradox: How CS Researchers Engage LLM Leaderboards](https://mlsanity.com/paper/2605.28966/) (2026)
- [Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs](https://mlsanity.com/paper/2606.17634/) (2026)
- [DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration](https://mlsanity.com/paper/2609.31215/) (2026)

## Abstract (arXiv)

Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences. Due to the high cost and time-consuming nature of human evaluations, an automatic LLM bencher (i.e., an automatic evaluation framework that aims to rank LLMs based on their alignment with human preferences) is indispensable. An automatic LLM bencher consists of four components: the input set (e.g., a user instruction), the evaluation model (e.g., an LLM), the evaluation type (e.g., pairwise comparison), and the aggregation method (e.g., the ELO rating system). However, previous work has not thoroughly explored how to select these components or how their different combinations influence the results. In this work, through controlled experiments, we provide a series of recommendations on how to choose each component to better automate the evaluation of LLMs. Furthermore, we discovered that when evaluating LLMs with similar performance, the performance of the automatic LLM bencher declines sharply, underscoring the limitations of current benchers and calling for future work. Lastly, we found that the evaluation models' performance at the instance level (e.g., the accuracy of selecting the best output) does not always align with their effectiveness when used as a component of a bencher, highlighting the importance of dedicated system-level evaluation of benchers.