mlsanityData to 14 Oct 2026

Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems

arXiv 2512.11150 ↗First posted 11 Dec 2025stat.ME stat.AP stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Measuring long-run LLM outcomes (user satisfaction, expert judgment, downstream KPIs) is expensive. Teams default to cheap LLM judges, but uncalibrated proxies can invert rankings entirely. Causal Judge Evaluation (CJE) makes it affordable to aim at the right target: calibrate cheap scores against a small oracle slice, then evaluate at scale with valid uncertainty. We treat surrogate validity as auditable: for each policy or deployment context, a small oracle audit tests whether the learned calibration remains mean-unbiased, turning an uncheckable identification condition into a falsifiable diagnostic. On 4,961 Chatbot Arena prompts comparing five policies with a 16x oracle/judge cost ratio, at a 5% oracle fraction CJE achieves 99% pairwise ranking accuracy at 14x lower cost; across all configurations (5-50% oracle, varying n), accuracy averages 94%. An adversarial policy fails the transport audit and is correctly flagged; in such cases CJE refuses level claims rather than reporting biased estimates. Key findings: naive confidence intervals on raw judge scores achieve 0% coverage (CJE: ~95%); importance-weighted estimators fail despite >90% effective sample size; and the Coverage-Limited Efficiency (CLE) bound and its TTC diagnostic explain why.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 68 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

19 of 68 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  8. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  9. 2015External Validity: From Do-Calculus to Transportability Across Populationscited by 75
  10. 2016Balancing Covariates via Propensity Score Weightingcited by 65
  11. 2017Balanced Policy Evaluation and Learningcited by 52
  12. 2020On the role of surrogates in the efficient estimation of treatment effects with limited outcome datacited by 38
7 more
  1. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  2. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 30
  3. 2013A General Algorithm for Deciding Transportability of Experimental Resultscited by 17
  4. 2025LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluationscited by 15
  5. 2024Doubly robust inference via calibrationcited by 9
  6. 2024Stabilized Inverse Probability Weighting via Isotonic Calibrationcited by 9
  7. 2025Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personascited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 9
  2. 2026A Finite-Calibration Regime Map for LLM Judge Panelscited by 2
  3. 2026Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Modelscited by 0