# Validating LLM-as-a-Judge Systems under Rating Indeterminacy

arXiv: https://arxiv.org/abs/2503.05965 · HTML: https://mlsanity.com/paper/2503.05965/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2025.

## Built on by (most cited first, 25 of 25)

- [Correlated Errors in Large Language Models](https://mlsanity.com/paper/2506.07962/) (2025)
- [Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale](https://mlsanity.com/paper/2601.03444/) (2026)
- [Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory](https://mlsanity.com/paper/2602.00521/) (2026)
- [Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges](https://mlsanity.com/paper/2605.06161/) (2026)
- [Reasoning LLMs are Wandering Solution Explorers](https://mlsanity.com/paper/2505.20296/) (2025)
- [Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why](https://mlsanity.com/paper/2606.00093/) (2026)
- [FALCON: Transforming Cyber Threat Intelligence into Deployable IDS Rules with Self-Reflection](https://mlsanity.com/paper/2508.18684/) (2025)
- [Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Risks](https://mlsanity.com/paper/2603.05801/) (2026)
- [ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents](https://mlsanity.com/paper/2606.00341/) (2026)
- [Discretizing Reward Models](https://mlsanity.com/paper/2606.21795/) (2026)
- [Enhancing LLM-Based Data Annotation with Error Decomposition](https://mlsanity.com/paper/2601.11920/) (2026)
- [CAFE: A Compound-AI Factorial Evaluation Framework](https://mlsanity.com/paper/2607.10380/) (2026)
- [Conduct Under Pressure: What Sixty Language Models Do When a User Pushes](https://mlsanity.com/paper/2609.25447/) (2026)
- [Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions](https://mlsanity.com/paper/2511.03047/) (2025)
- [Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation](https://mlsanity.com/paper/2604.03257/) (2026)
- [GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human](https://mlsanity.com/paper/2605.28882/) (2026)
- [NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision](https://mlsanity.com/paper/2607.08961/) (2026)
- [ASSERT: A Measurement Pipeline for GenAI Audits](https://mlsanity.com/paper/2608.13840/) (2026)
- [Door-in-the-Face Requests and Refusal Behaviour in Large Language Models](https://mlsanity.com/paper/2609.02707/) (2026)
- [LLJ Cards: Best practices for the Use of LLMs as Judges](https://mlsanity.com/paper/2609.24516/) (2026)
- [LLM Judge Validation Under Sparse Overlap: From Inference to Design](https://mlsanity.com/paper/2609.31857/) (2026)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)
- [VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision](https://mlsanity.com/paper/2610.00666/) (2026)
- [Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement](https://mlsanity.com/paper/2610.02492/) (2026)
- [SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges](https://mlsanity.com/paper/2610.19992/) (2026)

## Abstract (arXiv)

The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, plays a critical role in scaling and standardizing GenAI evaluations. To validate such judge systems, evaluators assess human--judge agreement by first collecting multiple human ratings for each item in a validation corpus, then aggregating the ratings into a single, per-item gold label rating. For many items, however, rating criteria may admit multiple valid interpretations, so a human or LLM rater may deem multiple ratings "reasonable" or "correct." We call this condition rating indeterminacy. Problematically, many rating tasks that contain rating indeterminacy rely on forced-choice elicitation, whereby raters are instructed to select only one rating for each item. In this paper, we introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy. We draw theoretical connections between different measures of judge system performance under different human--judge agreement metrics, and different rating elicitation and aggregation schemes. We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias LLM-as-a-judge validation. Through extensive experiments involving 11 real-world rating tasks and 9 commercial LLMs, we show that standard validation approaches that rely upon forced-choice ratings select judge systems that are highly suboptimal, performing as much as 31% worse than judge systems selected by our approach that uses multi-label "response set" ratings to account for rating indeterminacy. We conclude with concrete recommendations for more principled approaches to LLM-as-a-judge validation.