mlsanityData to 14 Oct 2026

The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction

arXiv 2608.07517 ↗First posted 2 Jul 2026cs.HC cs.CL stat.APcited by 0

Not on the map. Why?

Abstract (arXiv)

Can a multimodal LLM predict which version of a web page will win a real A/B test from screenshots alone? We report the most complete answer we are aware of, from six weeks of pre-registered experiments on real conversion tests: mostly no -- and the exceptions are identifiable in advance. On 330 real A/B tests a Gemini 3 Flash judge reaches Cohen's kappa = 0.14, but on the trustworthy (statistically significant) half of the labels the evidence is inconclusive (kappa = 0.11, CI includes zero). We show that 44% of the "ground-truth" labels in a leading CRO agency's catalog come from non-significant tests, and that the judge agrees more with the unreliable labels than the reliable ones -- a shared prior between labeler and model, not prediction. Every standard improvement lever (a 2.8x more expensive frontier model, prompt redesign, stimulus fidelity, change-type priors) fails its pre-registered gate. The judge's confident calls are different: a vote-margin gate isolates a subset (49% coverage) reaching kappa = 0.31 on significant labels. We measure the mechanism directly -- judges differing in model or prompt agree with each other at kappa = 0.74-0.88 while agreeing with real outcomes at only ~0.2, so a 16-vote panel carries about 2 effective independent votes -- and we reproduce it in humans: 15 CRO experts agree with each other (inter-rater kappa = 0.53) but score at chance against real outcomes (kappa ~ 0). Consensus, human or model, is reproducible, persuasive, and not evidence. We release our pre-registrations, locked gates, negative results, statistical harness, human responses, and a claims ledger in which every number carries an evidence tier.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 21 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

9 of 21 references.

References · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2017Selective Classification for Deep Neural Networkscited by 852
  7. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  8. 2020Selective Question Answering under Domain Shiftcited by 238
  9. 2025UXAgent: An LLM Agent-Based Usability Testing Framework for Web Designcited by 40

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026No Universal Signal Predicts Sample-Level LLM Regression under Version Updates
  2. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  3. 2026Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
  4. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
  5. 2026Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
  6. 2026Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
  7. 2025Reference-Free Rating of LLM Responses via Latent Information
  8. 2026Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction