# Monitoring Machine Learning Models: Online Detection of Relevant Deviations

arXiv: https://arxiv.org/abs/2309.15187 · HTML: https://mlsanity.com/paper/2309.15187/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Learning under Concept Drift: A Review](https://mlsanity.com/paper/2004.05785/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [AI Lifecycle Models Need To Be Revised. An Exploratory Study in Fintech](https://mlsanity.com/paper/2010.02716/) (2020)
- [Monitoring and explainability of models in production](https://mlsanity.com/paper/2007.06299/) (2020)
- [Sequential change-point detection based on nearest neighbors](https://mlsanity.com/paper/1604.03611/) (2016)
- [A Human-Centric Take on Model Monitoring](https://mlsanity.com/paper/2206.02868/) (2022)
- [Are deviations in a gradually varying mean relevant? A testing approach based on sup-norm estimators](https://mlsanity.com/paper/2002.06143/) (2020)
- [Testing relevant hypotheses in functional time series via self-normalization](https://mlsanity.com/paper/1809.06092/) (2018)
- [Page's Sequential Procedure for Change-Point Detection in Time Series Regression](https://mlsanity.com/paper/1308.1237/) (2013)

## Built on by (most cited first, 3 of 3)

- [OML-AD: Online Machine Learning for Anomaly Detection in Time Series Data](https://mlsanity.com/paper/2409.09742/) (2024)
- [Feature Engineering for Agents: An Adaptive Cognitive Architecture for Interpretable ML Monitoring](https://mlsanity.com/paper/2506.09742/) (2025)
- [KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement](https://mlsanity.com/paper/2608.02351/) (2026)

## Abstract (arXiv)

Machine learning models are essential tools in various domains, but their performance can degrade over time due to changes in data distribution or other factors. On one hand, detecting and addressing such degradations is crucial for maintaining the models' reliability. On the other hand, given enough data, any arbitrary small change of quality can be detected. As interventions, such as model re-training or replacement, can be expensive, we argue that they should only be carried out when changes exceed a given threshold. We propose a sequential monitoring scheme to detect these relevant changes. The proposed method reduces unnecessary alerts and overcomes the multiple testing problem by accounting for temporal dependence of the measured model quality. Conditions for consistency and specified asymptotic levels are provided. Empirical validation using simulated and real data demonstrates the superiority of our approach in detecting relevant changes in model quality compared to benchmark methods. Our research contributes a practical solution for distinguishing between minor fluctuations and meaningful degradations in machine learning model performance, ensuring their reliability in dynamic environments.