# Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

arXiv: https://arxiv.org/abs/2606.19544 · HTML: https://mlsanity.com/paper/2606.19544/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [RewardBench 2: Advancing Reward Model Evaluation](https://mlsanity.com/paper/2506.01937/) (2025)
- [Split and Merge: Aligning Position Biases in LLM-based Evaluators](https://mlsanity.com/paper/2310.01432/) (2023)
- [Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge](https://mlsanity.com/paper/2412.12509/) (2024)
- [Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks](https://mlsanity.com/paper/2510.27106/) (2025)
- [Improving LLM-as-a-Judge Inference with the Judgment Distribution](https://mlsanity.com/paper/2503.03064/) (2025)
- [How Reliable is Multilingual LLM-as-a-Judge?](https://mlsanity.com/paper/2505.12201/) (2025)
- [Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement](https://mlsanity.com/paper/2510.09738/) (2025)
- [Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic](https://mlsanity.com/paper/2512.08121/) (2025)

## Built on by (most cited first, 28 of 28)

- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training](https://mlsanity.com/paper/2609.04094/) (2026)
- [A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation](https://mlsanity.com/paper/2608.24419/) (2026)
- [Compaction as Epistemic Failure: How Agentic LLM Tools Fabricate Confirmed Results from Killed Processes](https://mlsanity.com/paper/2607.13071/) (2026)
- [Conduct Under Pressure: What Sixty Language Models Do When a User Pushes](https://mlsanity.com/paper/2609.25447/) (2026)
- [NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision](https://mlsanity.com/paper/2607.08961/) (2026)
- [Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers](https://mlsanity.com/paper/2608.26762/) (2026)
- [CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Science](https://mlsanity.com/paper/2608.28631/) (2026)
- [Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints](https://mlsanity.com/paper/2609.04198/) (2026)
- [Chronicle: Cut-Point Replay for Regression Testing of LLM Agents](https://mlsanity.com/paper/2609.20625/) (2026)
- [Self-Cleaning and Captured Anyway: One Measured Primitive for Error in a Store an Agent Writes to Itself, and What a Falling Score Actually Measures](https://mlsanity.com/paper/2609.25052/) (2026)
- [Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms](https://mlsanity.com/paper/2609.27321/) (2026)
- [Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM](https://mlsanity.com/paper/2609.27787/) (2026)
- [Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents](https://mlsanity.com/paper/2609.37236/) (2026)
- [A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses](https://mlsanity.com/paper/2609.37788/) (2026)
- [The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating](https://mlsanity.com/paper/2610.00054/) (2026)
- [When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora](https://mlsanity.com/paper/2610.00202/) (2026)
- [Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation](https://mlsanity.com/paper/2610.01670/) (2026)
- [A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined](https://mlsanity.com/paper/2610.01938/) (2026)
- [IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation](https://mlsanity.com/paper/2610.04074/) (2026)
- [Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data](https://mlsanity.com/paper/2610.04370/) (2026)
- [TrustMI: Causally controlling how assistants trust their users](https://mlsanity.com/paper/2610.06064/) (2026)
- [Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations](https://mlsanity.com/paper/2610.08364/) (2026)
- [Language-model ratings of depression reflect the rater more than the patient](https://mlsanity.com/paper/2610.08501/) (2026)
- [LLM Persuasion Is in the Eye of the Evaluation](https://mlsanity.com/paper/2610.10232/) (2026)
- [The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Much](https://mlsanity.com/paper/2610.15938/) (2026)
- [The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express](https://mlsanity.com/paper/2610.17158/) (2026)
- [SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges](https://mlsanity.com/paper/2610.19992/) (2026)

## Abstract (arXiv)

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.