mlsanityData to 14 Oct 2026

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

arXiv 2606.19544 ↗First posted 17 Jun 2026cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

26 of 40 references.

References · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  8. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
14 more
  1. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  2. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  4. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  5. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  6. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  7. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  8. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 72
  9. 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 56
  10. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  11. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  12. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  13. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 25
  14. 2025Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statisticcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 28.

28 papers.

Sort:

By area:

  1. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  2. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 5
  3. 2026A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluationcited by 4
  4. 2026Compaction as Epistemic Failure: How Agentic LLM Tools Fabricate Confirmed Results from Killed Processescited by 1
  5. 2026Conduct Under Pressure: What Sixty Language Models Do When a User Pushescited by 1
  6. 2026NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervisioncited by 0
  7. 2026Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorerscited by 0
  8. 2026CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Sciencecited by 0
  9. 2026Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpointscited by 0
  10. 2026Chronicle: Cut-Point Replay for Regression Testing of LLM Agentscited by 0
  11. 2026Self-Cleaning and Captured Anyway: One Measured Primitive for Error in a Store an Agent Writes to Itself, and What a Falling Score Actually Measurescited by 0
  12. 2026Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanismscited by 0
16 more
  1. 2026Ask Which, Not How Good: Sizing Benchmarks Scored by an LLMcited by 0
  2. 2026Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agentscited by 0
  3. 2026A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responsescited by 0
  4. 2026The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generatingcited by 0
  5. 2026When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corporacited by 0
  6. 2026Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservationcited by 0
  7. 2026A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combinedcited by 0
  8. 2026IdeaScientist: Orchestrating Agents for Grounded Scientific Ideationcited by 0
  9. 2026Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Datacited by 0
  10. 2026TrustMI: Causally controlling how assistants trust their userscited by 0
  11. 2026Transect: Retaining Observability for Long-Horizon LLM Agent Evaluationscited by 0
  12. 2026Language-model ratings of depression reflect the rater more than the patientcited by 0
  13. 2026LLM Persuasion Is in the Eye of the Evaluationcited by 0
  14. 2026The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Muchcited by 0
  15. 2026The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Expresscited by 0
  16. 2026SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judgescited by 0