mlsanityData to 14 Oct 2026

Can LLMs Rank? A Tale of Triads and Triage

arXiv 2606.30412 ↗First posted 29 Jun 2026cs.CY cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency zeta, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's τ, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 93 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

19 of 93 references.

References · 19

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  5. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
  6. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  7. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 34
  8. 2015Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issuescited by 32
  9. 2021Fairness Perceptions of Algorithmic Decision-Making: A Systematic Review of the Empirical Literaturecited by 27
  10. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  11. 2024LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Rankingcited by 16
  12. 2023Algorithmic Harms in Child Welfare: Uncertainties in Practice, Organization, and Street-level Decision-Makingcited by 12
7 more
  1. 2024A Statistical Framework for Ranking LLM-Based Chatbotscited by 12
  2. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  3. 2023Discretionary Trees: Understanding Street-Level Bureaucracy via Machine Learningcited by 5
  4. 2025ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstructioncited by 5
  5. 2025Street-Level AI: Are Large Language Models Ready for Real-World Judgments?cited by 3
  6. 2026Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violationscited by 3
  7. 2024Analysis of the inference of ratings and rankings in complex networks using discrete exterior calculus on higher--order networkscited by 1

Built on by

Not cited yet.