mlsanityData to 14 Oct 2026

Sequential Harmful Shift Detection Without Labels

arXiv 2412.12910 ↗First posted 17 Dec 2024stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We introduce a novel approach for detecting distribution shifts that negatively impact the performance of machine learning models in continuous production environments, which requires no access to ground truth data labels. It builds upon the work of Podkopaev and Ramdas [2022], who address scenarios where labels are available for tracking model errors over time. Our solution extends this framework to work in the absence of labels, by employing a proxy for the true error. This proxy is derived using the predictions of a trained error estimator. Experiments show that our method has high power and false alarm control under various distribution shifts, including covariate and label shifts and natural shifts over geography and time.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

20 of 34 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  5. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 449
  6. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
  7. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 229
  8. 2020Estimating means of bounded random variables by bettingcited by 169
  9. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  10. 2021Predicting with Confidence on Unseen Distributionscited by 116
  11. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  12. 2022From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectorscited by 83
8 more
  1. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  2. 2024Active Statistical Inferencecited by 50
  3. 2021Tracking the risk of a deployed model and detecting harmful distribution shiftscited by 40
  4. 2015Always Valid Inference: Bringing Sequential Analysis to A/B Testingcited by 20
  5. 2022A Learning Based Hypothesis Test for Harmful Covariate Shiftcited by 19
  6. 2023Adaptive Conformal Prediction by Reweighting Nonconformity Scorecited by 16
  7. 2018A Bandit Approach to Multiple Testing with False Discovery Controlcited by 7
  8. 2022Predicting Out-of-Domain Generalization with Neighborhood Invariancecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 17
  2. 2025WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingalescited by 16
  3. 2025Monitoring Risks in Test-Time Adaptationcited by 11
  4. 2025On Continuous Monitoring of Risk Violations under Unknown Shiftcited by 10
  5. 2025Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradationcited by 6
  6. 2026Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shiftscited by 3
  7. 2026Active Multiple-Prediction-Powered Inferencecited by 1
  8. 2026Label-Efficient Monitoring of Classification Models via Stratified Importance Samplingcited by 0
  9. 2026Online Safety Monitoring for LLMscited by 0
  10. 2026The Premise Is the Problem: Exchangeability Failure in Self-Monitored Test-Time Adaptationcited by 0