mlsanityData to 14 Oct 2026

Investigating Non-Transitivity in LLM-as-a-Judge

arXiv 2502.14074 ↗First posted 19 Feb 2025cs.AI cs.CL cs.LGcited by 34

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the assumption of transitive preferences. However, the validity of this assumption remains largely unexplored. In this study, we investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings. We find that LLM judges exhibit non-transitive preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-Terry models of preference can produce more reliable rankings. Notably, our method increases both the Spearman correlation and the Kendall correlation with Chatbot Arena (95.0% -> 96.4% and 82.1% -> 86.3% respectively). To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments, using a dynamic matching strategy to capture the benefits of round-robin tournaments while maintaining computational efficiency.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 45 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

26 of 45 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  10. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  11. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
14 more
  1. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  2. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  3. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  5. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  6. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  7. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  8. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  9. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  10. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  11. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  12. 2020Real World Games Look Like Spinning Topscited by 66
  13. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  14. 2024A Generalized Model for Multidimensional Intransitivitycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 19.

34 papers.

Sort:

By area:

  1. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  2. 2026Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scalecited by 20
  3. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
  4. 2025Generative Value Conflicts Reveal LLM Prioritiescited by 13
  5. 2025Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Modelscited by 10
  6. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 10
  7. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  8. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  9. 2025Skewed Score: A statistical framework to assess autograderscited by 7
  10. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 7
  11. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 6
  12. 2026Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoningcited by 6
22 more
  1. 2025ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstructioncited by 5
  2. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  3. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
  4. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  5. 2026Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluationcited by 3
  6. 2025Matching Markets Meet LLMs: Algorithmic Reasoning with Ranked Preferencescited by 2
  7. 2025Knowing the Facts but Choosing the Shortcut: Understanding How Large Language Models Compare Entitiescited by 2
  8. 2025Fairness Evaluation and Inference Level Mitigation in LLMscited by 2
  9. 2026Causal methods for LLM development and evaluationcited by 2
  10. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 2
  11. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluationcited by 2
  12. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  13. 2026Scaling Reward Modeling without Human Supervisioncited by 1
  14. 2026PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluationcited by 1
  15. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  16. 2026Soft Tournament Equilibrium: Differentiable Set-Valued Inference for Non-Transitive Pairwise Comparisonscited by 0
  17. 2026Generalized Priority-Aware Shapley Valuecited by 0
  18. 2026Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphscited by 0
  19. 2026To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregationcited by 0
  20. 2026Can LLMs Rank? A Tale of Triads and Triagecited by 0
  21. 2026Multiple latent orderings better predict language model preferencescited by 0
  22. 2026Can You Trust Your Preference Models? Stackelberg Games for Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  2. 2025Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness
  3. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judges
  5. 2025SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges
  6. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
  7. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors
  8. 2026JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols