When Accuracy Gaps Fail to Certify: Auditing Cross-Domain Recalibration of LLM Judges
Not on the map. Why?
Abstract (arXiv)
A scalar recalibration map fitted for an LLM judge on one task can fail when the task distribution changes, but the source-target accuracy gap is often treated as a proxy for that failure. We test what this gap can predict and what it can certify across thirteen judges, two generators, eight domains, and 1,176 predeclared transfers. After accounting for mean score shift, the gap yields a population lower bound on target calibration error, yet identical gaps can induce opposite transfer outcomes. Exact importance weighting recovers target proper loss under covariate shift, so failure of an estimated weighting pipeline does not by itself establish conditional shift. A finite-sample simultaneous lower certificate converts the population bound into a one-sided rejection rule using audit labels disjoint from evaluation outcomes. The leak-free gap correlation is 0.25 (95% CI [-0.09, 0.55]), falls to 0.09 on the second generator, and does not support a generator-invariant association. The certificate retains nominal coverage but has power 0.13 even at m=1024, whereas target-domain temperature scaling with 16 labels reaches harm rate 0.09, compared with 0.34 for source-fitted Platt scaling. Accuracy gaps are therefore weak warning signals for scalar probability transfer, not deployment certificates.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 27 references that are in the map. First posted on arXiv 23 Aug 2026.
Builds on
12 of 27 references.
References · 12
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2020Mitigating Bias in Calibration Error Estimationcited by 101
- 2021Distribution-free uncertainty quantification for classification under label shiftcited by 101
- 2020Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptationcited by 43
- 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 27
- 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
- 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 18
Built on by
Not cited yet.