mlsanityData to 14 Oct 2026

Decomposing LLM-Judge Uncertainty to Target Expert Labels

arXiv 2609.06444 ↗First posted 6 Sep 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there. We demonstrate we can estimate where a judge is ignorant rather than where experts genuinely disagree, and propose using this to direct expert labelling. Code and data are available at https://github.com/composo-ai/judge-uncertainty-decomposition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 16 references that are in the map. First posted on arXiv 6 Sep 2026.

Builds on

11 of 16 references.

References · 11

  1. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  3. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  4. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  5. 2024From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimationcited by 28
  6. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 27
  7. 2019Epistemic Uncertainty Samplingcited by 24
  8. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
  9. 2025LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Taskcited by 9
  10. 2026LLM Performance Predictors: Learning When to Escalate in Hybrid Human-AI Moderation Systemscited by 4
  11. 2026On Cost-Effective LLM-as-a-Judge Improvement Techniquescited by 3

Built on by

Not cited yet.