mlsanityData to 14 Oct 2026

Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge

arXiv 2610.14029 ↗First posted 30 Sep 2026cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

27 of 38 references.

References · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  9. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  11. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
15 more
  1. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  2. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  3. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  4. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 113
  5. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  6. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  7. 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 56
  8. 2024Bayesian Calibration of Win Rate Estimation with LLM Evaluatorscited by 16
  9. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 16
  10. 2024A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimatorscited by 15
  11. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
  12. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
  13. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 9
  14. 2024Data Quality in Crowdsourcing and Spamming Behavior Detectioncited by 2
  15. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guaranteescited by 2

Built on by

Not cited yet.