mlsanityData to 14 Oct 2026

MLDemon: Deployment Monitoring for Machine Learning Systems

arXiv 2104.13621 ↗First posted 28 Apr 2021cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

Post-deployment monitoring of ML systems is critical for ensuring reliability, especially as new user inputs can differ from the training distribution. Here we propose a novel approach, MLDemon, for ML DEployment MONitoring. MLDemon integrates both unlabeled data and a small amount of on-demand labels to produce a real-time estimate of the ML model's current performance on a given data stream. Subject to budget constraints, MLDemon decides when to acquire additional, potentially costly, expert supervised labels to verify the model. On temporal datasets with diverse distribution drifts and models, MLDemon outperforms existing approaches. Moreover, we provide theoretical analysis to show that MLDemon is minimax rate optimal for a broad class of distribution drifts.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 14 of its 66 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

14 of 66 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Learning under Concept Drift: A Reviewcited by 477
  3. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
  4. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 243
  5. 2020A Unified View of Label Shift Estimationcited by 119
  6. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  7. 2019Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysiscited by 92
  8. 2018Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLcited by 81
  9. 2020Masked Face Recognition Dataset and Applicationcited by 66
  10. 2021Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Aheadcited by 41
  11. 2020FrugalML: How to Use ML Prediction APIs More Accurately and Cheaplycited by 33
  12. 2018Request-and-Reverify: Hierarchical Hypothesis Testing for Concept Drift Detection with Expensive Labelscited by 11
2 more
  1. 2019Automatic Model Monitoring for Data Streamscited by 9
  2. 2020Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoringcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 4; 2025: 3.

14 papers.

Sort:

  1. 2022Calibrated Selective Classificationcited by 33
  2. 2022GSCLIP : A Framework for Explaining Distribution Shifts in Natural Languagecited by 14
  3. 2024Confidence-based Estimators for Predictive Performance in Model Monitoringcited by 8
  4. 2025Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradationcited by 6
  5. 2023An investigation of challenges encountered when specifying training data and runtime monitors for safety critical ML applicationscited by 4
  6. 2024Randomized Confidence Bounds for Stochastic Partial Monitoringcited by 3
  7. 2023MLHOps: Machine Learning for Healthcare Operationscited by 2
  8. 2023Adversarial Learning for Feature Shift Detection and Correctioncited by 2
  9. 2024A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validationcited by 1
  10. 2025Self-Supervised Data Generation for Precision Agriculture: Blending Simulated Environments with Real Imagerycited by 1
  11. 2022Active Learning Framework to Automate NetworkTraffic Classificationcited by 0
  12. 2022An Efficient Framework for Monitoring Subgroup Performance of Machine Learning Systemscited by 0
2 more
  1. 2024Combining Embeddings and Domain Knowledge for Job Posting Duplicate Detectioncited by 0
  2. 2025Reliably Detecting Model Failures in Deployment Without Labelscited by 0