Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Not on the map. Why?
Abstract (arXiv)
LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: (1) a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (barρ = 0.8-4.1%), with 33-67% of documents exhibiting at least one directed 3-cycle; and (2) split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed ≥(1-α) coverage, with set width serving as a per-instance reliability indicator (rs = +0.576, N=1,918, p < 10-100, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (barr = 0.32-0.38), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size ≈ 3.0) and coherence moderately so (avg. set size ≈ 3.9), while fluency and consistency remain unreliable (avg. set size ≈ 4.9). We release all code, prompts, and cached results.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 16 Apr 2026.
Builds on
16 of 26 references.
References · 16
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
- 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
- 2019Conformal Prediction Under Covariate Shiftcited by 519
- 2023GPTScore: Evaluate as You Desirecited by 361
- 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
- 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
- 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
- 2023Conformal Language Modelingcited by 170
- 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
4 more
- 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
- 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
- 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
- 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
Built on by
Citing papers per year: 2026: 3.
3 papers.