mlsanityData to 13 Oct 2026

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

arXiv 2606.15474 ↗First posted 13 Jun 2026cs.AI stat.APcited by 8

Not on the map. Why?

Abstract (arXiv)

Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

17 of 31 references.

References · 17

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  2. 2020Learning to summarize from human feedbackcited by 2,575
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 233
  4. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 224
  5. 2020Estimating means of bounded random variables by bettingcited by 166
  6. 2022E-detectors: a nonparametric framework for sequential change detectioncited by 30
  7. 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 26
  8. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  9. 2025Prediction-Powered E-Valuescited by 16
  10. 2025Anytime-valid, Bayes-assisted, Prediction-Powered Inferencecited by 7
  11. 2026Revisiting Active Sequential Prediction-Powered Mean Estimationcited by 4
  12. 2026Bias and Uncertainty in LLM-as-a-Judge Estimationcited by 4
5 more
  1. 2026Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shiftscited by 3
  2. 2026Power Analysis for Prediction-Powered Inferencecited by 3
  3. 2025Extending Prediction-Powered Inference through Conformal Predictioncited by 2
  4. 2024Round Robin Active Sequential Change Detection for Dependent Multi-Channel Datacited by 1
  5. 2026Drift-to-Action Controllers: Budgeted Interventions with Online Risk Certificatescited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 8.

8 papers.