Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
Not on the map. Why?
Abstract (arXiv)
Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In practice, LLM judges are often trained and evaluated in similar ways, so they can make the same mistakes. We study how this dependency affects the reliability of consensus. We find substantial error correlation across both open-weight and frontier LLM judges. In our main bank of ten judges, the average pairwise correlation between judge errors is 0.21. As a result, the ten judges only provide roughly as much statistical information as 3.5 independent judges. The dependency is even stronger among the high-accuracy frontier judges we evaluate, including judges from different providers. In up to 28% of our comparisons, ignoring shared errors leads to the conclusion that one system is significantly better, while accounting for them does not. We also find that the pattern of errors matters. Errors shared by most judges and errors concentrated among a smaller group affect consensus differently and favor different voting methods. Measuring the overall amount of correlation alone is therefore insufficient. Our results suggest a simple approach: use a small set of trusted examples to estimate judge accuracy and identify shared mistakes. These shared errors should then be considered when analyzing the results, and the voting method should be chosen using trusted examples before it is applied to new data.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
22 of 28 references.
References · 22
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
- 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
- 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
10 more
- 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
- 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
- 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
- 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
- 2025One Token to Fool LLM-as-a-Judgecited by 60
- 2025Great Models Think Alike and this Undermines AI Oversightcited by 58
- 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
- 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
- 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
- 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 18
Built on by
Citing papers per year: 2026: 1.
1 paper.