# Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

arXiv: https://arxiv.org/abs/2605.29800 · HTML: https://mlsanity.com/paper/2605.29800/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 18 of 18)

- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)](https://mlsanity.com/paper/2510.22954/) (2025)
- [Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement](https://mlsanity.com/paper/2407.18370/) (2024)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data](https://mlsanity.com/paper/2410.13341/) (2024)
- [Neither Valid nor Reliable? Investigating the Use of LLMs as Judges](https://mlsanity.com/paper/2508.18076/) (2025)
- [Can Large Language Models Capture Dissenting Human Voices?](https://mlsanity.com/paper/2305.13788/) (2023)
- [Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection](https://mlsanity.com/paper/2602.08003/) (2026)
- [Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?](https://mlsanity.com/paper/2506.19467/) (2025)
- [Bridging Human and LLM Judgments: Understanding and Narrowing the Gap](https://mlsanity.com/paper/2508.12792/) (2025)
- [Examining Independence in Ensemble Sentiment Analysis: A Study on the Limits of Large Language Models Using the Condorcet Jury Theorem](https://mlsanity.com/paper/2409.00094/) (2024)

## Built on by (most cited first, 26 of 26)

- [When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling](https://mlsanity.com/paper/2606.28661/) (2026)
- [Memory Reward Inflation in Self-Improving LLM Agents](https://mlsanity.com/paper/2608.00017/) (2026)
- [A Finite-Calibration Regime Map for LLM Judge Panels](https://mlsanity.com/paper/2606.01034/) (2026)
- [Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings](https://mlsanity.com/paper/2607.13918/) (2026)
- [Stopping and Routing LLM Judge Panels](https://mlsanity.com/paper/2608.19802/) (2026)
- [Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts](https://mlsanity.com/paper/2609.06386/) (2026)
- [JuryProbe: An Empirical Consensus-Risk Diagnostic for Routing Reference-Free Factuality Judge Panels to Grounded Verification](https://mlsanity.com/paper/2608.20607/) (2026)
- [Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels](https://mlsanity.com/paper/2609.17857/) (2026)
- [A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memory](https://mlsanity.com/paper/2609.30813/) (2026)
- [SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation](https://mlsanity.com/paper/2607.07469/) (2026)
- [Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps](https://mlsanity.com/paper/2608.06940/) (2026)
- [Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification](https://mlsanity.com/paper/2608.07920/) (2026)
- [Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees](https://mlsanity.com/paper/2608.08002/) (2026)
- [The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering](https://mlsanity.com/paper/2608.21262/) (2026)
- [Aggregate Disambiguation Systems](https://mlsanity.com/paper/2608.30805/) (2026)
- [A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model](https://mlsanity.com/paper/2609.08826/) (2026)
- [Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures](https://mlsanity.com/paper/2609.13463/) (2026)
- [A latent dimension of Condorcet's jury theorem for multiple AI advisers](https://mlsanity.com/paper/2609.14438/) (2026)
- [Who Audits Whom, on What Substrate, with What Evidence? An Independence-Graded Audit Protocol for Agentic AI](https://mlsanity.com/paper/2609.18272/) (2026)
- [Unanimity Without Persuasion: A Single Round of Debate Erases the Disagreement That Verification Needs](https://mlsanity.com/paper/2609.26145/) (2026)
- [Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges](https://mlsanity.com/paper/2609.27110/) (2026)
- [Binarization Flattens the Score Space](https://mlsanity.com/paper/2609.35797/) (2026)
- [RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection](https://mlsanity.com/paper/2609.39229/) (2026)
- [Copies or Sources? Measuring How LLM Aggregators Count Restated Evidence in Multi-Agent Systems](https://mlsanity.com/paper/2610.06192/) (2026)
- [Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?](https://mlsanity.com/paper/2610.07359/) (2026)
- [Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR](https://mlsanity.com/paper/2610.17824/) (2026)

## Abstract (arXiv)

LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.