Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Bounds
Not on the map. Why?
Abstract (arXiv)
Every production model is updated, by retraining, fine-tuning, quantization, or a silent vendor swap, and each update risks being worse than what it replaced. We formalize update promotion as certified paired risk-difference auditing. Our starting point is a support identity: the risk difference between two models lives on the inputs where they disagree, observable without labels. We build DISCERN, a sequential two-tier protocol. A zero-label tier certifies benign updates whose disagreement rate is below tolerance from unlabeled traffic alone. An audited tier labels only sampled disagreements through an anytime-valid confidence sequence, valid at every stopping time and under any label-routing rule, even an adversarial judge. We prove finite-sample validity and matching label-complexity bounds of order rho^2/eps^2 at the rate level, so exploiting free disagreement provably saves a factor 1/rho over any pairing-blind auditor, and the guarantee composes across an unbounded sequence of promotions from one error budget. Across 14,000+ replayed audit streams over 785 update pairs, including LoRA fine-tunes of language models up to 1.4B parameters, miscoverage is 0.0002 (nominal 5%), power 0.986 with zero false alarms, and 56% of benign updates certify with zero labels. Each audit emits a machine-checkable evidence record for post-market monitoring.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 22 Jul 2026.
Builds on
29 of 47 references.
References · 29
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 286
- 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 229
- 2023Prediction-Powered Inferencecited by 220
- 2020Estimating means of bounded random variables by bettingcited by 169
- 2019Predictive Multiplicity in Classificationcited by 166
- 2022Game-theoretic statistics and safe anytime-valid inferencecited by 155
- 2023PPI++: Efficient Prediction-Powered Inferencecited by 117
- 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
- 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 57
- 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 57
17 more
- 2024Active Statistical Inferencecited by 50
- 2021Tracking the risk of a deployed model and detecting harmful distribution shiftscited by 40
- 2020An Empirical Analysis of Backward Compatibility in Machine Learning Systemscited by 38
- 2023Auditing Fairness by Bettingcited by 25
- 2021Off-policy Confidence Sequencescited by 15
- 2024Active, anytime-valid risk controlling prediction setscited by 14
- 2026PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agentscited by 13
- 2026Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelinescited by 8
- 2026Resolution Diagnostics for Paired LLM Evaluationcited by 7
- 2026Cross-Model Disagreement as a Label-Free Correctness Signalcited by 6
- 2019Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creepcited by 4
- 2023Active Sequential Two-Sample Testingcited by 4
- 2025Evaluating multiple models using labeled and unlabeled datacited by 4
- 2026STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systemscited by 3
- 2026Adaptive auditing of AI systems with anytime-valid guaranteescited by 3
- 2025Advanced Tutorial: Label-Efficient Two-Sample Testscited by 2
- 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows
- 2026Phantom Gains: Auditing Self-Improvement Against a Measured Null
- 2026Phantom Merge: When Your Large Language Model Agents Pick One but Tell You About Another
- 2026Available Guardrails: Certifying Selective Prediction across ML Systems
- 2026Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation
- 2026Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch
- 2026When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels
- 2026Understanding Disclosure Risk in Differential Privacy with Applications to Noise Calibration and Auditing (Extended Version)