# Can LLMs Rank? A Tale of Triads and Triage

arXiv: https://arxiv.org/abs/2606.30412 · HTML: https://mlsanity.com/paper/2606.30412/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 93 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [Verbosity Bias in Preference Labeling by Large Language Models](https://mlsanity.com/paper/2310.10076/) (2023)
- [Improving LLM-as-a-Judge Inference with the Judgment Distribution](https://mlsanity.com/paper/2503.03064/) (2025)
- [Investigating Non-Transitivity in LLM-as-a-Judge](https://mlsanity.com/paper/2502.14074/) (2025)
- [Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues](https://mlsanity.com/paper/1510.05610/) (2015)
- [Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literature](https://mlsanity.com/paper/2103.12016/) (2021)
- [TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them](https://mlsanity.com/paper/2509.21117/) (2025)
- [LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking](https://mlsanity.com/paper/2406.00231/) (2024)
- [Algorithmic Harms in Child Welfare: Uncertainties in Practice, Organization, and Street-level Decision-Making](https://mlsanity.com/paper/2308.05224/) (2023)
- [A Statistical Framework for Ranking LLM-Based Chatbots](https://mlsanity.com/paper/2412.18407/) (2024)
- [Who can we trust? LLM-as-a-jury for Comparative Assessment](https://mlsanity.com/paper/2602.16610/) (2026)
- [Discretionary Trees: Understanding Street-Level Bureaucracy via Machine Learning](https://mlsanity.com/paper/2312.10694/) (2023)
- [ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstruction](https://mlsanity.com/paper/2505.17691/) (2025)
- [Street-Level AI: Are Large Language Models Ready for Real-World Judgments?](https://mlsanity.com/paper/2508.08193/) (2025)
- [Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations](https://mlsanity.com/paper/2604.15302/) (2026)
- [Analysis of the inference of ratings and rankings in complex networks using discrete exterior calculus on higher--order networks](https://mlsanity.com/paper/2411.02434/) (2024)

## Abstract (arXiv)

From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency zeta, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's τ, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.