# Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement

arXiv: https://arxiv.org/abs/2510.09738 · HTML: https://mlsanity.com/paper/2510.09738/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2407.11005/) (2024)

## Built on by (most cited first, 25 of 25)

- [Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias](https://mlsanity.com/paper/2606.19544/) (2026)
- [When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability](https://mlsanity.com/paper/2607.08535/) (2026)
- [Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness](https://mlsanity.com/paper/2601.08843/) (2025)
- [Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents](https://mlsanity.com/paper/2606.10315/) (2026)
- [A Systematic Investigation of Document Chunking Strategies and Embedding Sensitivity](https://mlsanity.com/paper/2603.06976/) (2026)
- [AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding](https://mlsanity.com/paper/2603.16934/) (2026)
- [Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations](https://mlsanity.com/paper/2510.20743/) (2025)
- [EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation](https://mlsanity.com/paper/2602.18823/) (2026)
- [Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization](https://mlsanity.com/paper/2604.20726/) (2026)
- [CAFE: A Compound-AI Factorial Evaluation Framework](https://mlsanity.com/paper/2607.10380/) (2026)
- [Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation](https://mlsanity.com/paper/2608.00794/) (2026)
- [Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF](https://mlsanity.com/paper/2603.21359/) (2026)
- [Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation](https://mlsanity.com/paper/2604.03257/) (2026)
- [Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigation](https://mlsanity.com/paper/2605.11730/) (2026)
- [TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching](https://mlsanity.com/paper/2605.12288/) (2026)
- [Persona-Conditioned Adversarial Prompting (PCAP): Multi-Identity Red-Teaming for Enhanced Adversarial Prompt Discovery](https://mlsanity.com/paper/2605.12565/) (2026)
- [Fully Open Meditron: An Auditable Pipeline for Clinical LLMs](https://mlsanity.com/paper/2605.16215/) (2026)
- [Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing](https://mlsanity.com/paper/2606.00686/) (2026)
- [LLM-as-a-Discriminator: When Synthetic Tables Still Look Real](https://mlsanity.com/paper/2606.09865/) (2026)
- [SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses](https://mlsanity.com/paper/2607.02049/) (2026)
- [A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models](https://mlsanity.com/paper/2607.21632/) (2026)
- [Can LLMs Normalize Databases? A Benchmark and Multi-Agent Framework for Schema Normalization](https://mlsanity.com/paper/2609.11141/) (2026)
- [Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline](https://mlsanity.com/paper/2609.30290/) (2026)
- [CORA: A Protocol for Diagnosing Boundary Robustness in Text-to-Audio Retrieval under Query Reformulations](https://mlsanity.com/paper/2609.33433/) (2026)
- [RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection](https://mlsanity.com/paper/2609.39229/) (2026)

## Abstract (arXiv)

This research introduces the Judge's Verdict Benchmark, a novel two-step methodology to evaluate Large Language Models (LLMs) as judges for response accuracy evaluation tasks. We assess how well 54 LLMs can replicate human judgment when scoring responses from RAG (Retrieval-Augmented Generation) or Agentic pipelines against ground truth answers. Our methodology progresses from traditional correlation analysis to comprehensive Cohen's Kappa analysis that measures actual agreement patterns. The two-step approach includes: (1) a correlation test that filters judges with strong alignment, followed by (2) a human-likeness test using z-scores to identify two distinct judgment patterns: human-like judgment (|z| < 1) that mimics natural human variation, and super-consistent judgment (z > 1) that exceeds typical human-to-human agreement levels. This methodology reveals that 27 out of 54 tested LLMs achieve Tier 1 performance: 23 models exhibit human-like patterns that preserve the nuances of human judgment, while 4 models demonstrate super-consistent behavior, a pattern that could indicate either enhanced reliability or oversimplification of complex judgments. Testing 43 open-source models (1B-405B parameters) and 11 closed models (GPT, Gemini, Claude variants), we demonstrate that judge excellence is not solely dependent on model size but on specific training strategies. Our key contributions include: (1) establishing that correlation alone is insufficient for judge evaluation, (2) introducing a "Turing Test for judges" based on agreement patterns, and (3) providing a standardized benchmark for classifying LLM judges into distinct performance tiers for different evaluation needs.