Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Not on the map. Why?
Abstract (arXiv)
Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.
Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 13 Jun 2026.
Builds on
17 of 31 references.
References · 17
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2020Learning to summarize from human feedbackcited by 2,575
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 233
- 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 224
- 2020Estimating means of bounded random variables by bettingcited by 166
- 2022E-detectors: a nonparametric framework for sequential change detectioncited by 30
- 2025How to Correctly Report LLM-as-a-Judge Evaluationscited by 26
- 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
- 2025Prediction-Powered E-Valuescited by 16
- 2025Anytime-valid, Bayes-assisted, Prediction-Powered Inferencecited by 7
- 2026Revisiting Active Sequential Prediction-Powered Mean Estimationcited by 4
- 2026Bias and Uncertainty in LLM-as-a-Judge Estimationcited by 4
5 more
- 2026Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shiftscited by 3
- 2026Power Analysis for Prediction-Powered Inferencecited by 3
- 2025Extending Prediction-Powered Inference through Conformal Predictioncited by 2
- 2024Round Robin Active Sequential Change Detection for Dependent Multi-Channel Datacited by 1
- 2026Drift-to-Action Controllers: Budgeted Interventions with Online Risk Certificatescited by 1
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agentscited by 3
- 2026Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Boundscited by 2
- 2026Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?cited by 1
- 2026EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systemscited by 0
- 2026Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviewscited by 0
- 2026On Evaluating and Improving Conversational Agents in Productioncited by 0
- 2026When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Driftcited by 0
- 2026Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Bankingcited by 0
- 2026Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking30 Sep 2026
- 2026When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift26 Sep 2026
- 2026On Evaluating and Improving Conversational Agents in Production25 Sep 2026
- 2026Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviews8 Sep 2026
- 2026Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Bounds22 Jul 2026
- 2026EPC: A Standardized Protocol for Measuring Evaluator Preference Dynamics in LLM Agent Systems1 Jul 2026
- 2026Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?30 Jun 2026
- 2026A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents29 Jun 2026