# A monitoring framework for deployed machine learning models with supply chain examples

arXiv: https://arxiv.org/abs/2211.06239 · HTML: https://mlsanity.com/paper/2211.06239/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 29 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 4 of 4)

- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Monitoring and explainability of models in production](https://mlsanity.com/paper/2007.06299/) (2020)
- [Context-Aware Drift Detection](https://mlsanity.com/paper/2203.08644/) (2022)
- [Scalable Deployment of AI Time-series Models for IoT](https://mlsanity.com/paper/2003.12141/) (2020)

## Built on by (most cited first, 5 of 5)

- [DriftGuard: Safety-Aware Multi-Monitor Detection and Selective Adaptation for Evolving Toxicity Moderation](https://mlsanity.com/paper/2606.28725/) (2026)
- [Feature Engineering for Agents: An Adaptive Cognitive Architecture for Interpretable ML Monitoring](https://mlsanity.com/paper/2506.09742/) (2025)
- [A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practice](https://mlsanity.com/paper/2503.00563/) (2025)
- [Understanding Practitioners Perspectives on Monitoring Machine Learning Systems](https://mlsanity.com/paper/2509.25195/) (2025)
- [KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement](https://mlsanity.com/paper/2608.02351/) (2026)

## Abstract (arXiv)

Actively monitoring machine learning models during production operations helps ensure prediction quality and detection and remediation of unexpected or undesired conditions. Monitoring models already deployed in big data environments brings the additional challenges of adding monitoring in parallel to the existing modelling workflow and controlling resource requirements. In this paper, we describe (1) a framework for monitoring machine learning models; and, (2) its implementation for a big data supply chain application. We use our implementation to study drift in model features, predictions, and performance on three real data sets. We compare hypothesis test and information theoretic approaches to drift detection in features and predictions using the Kolmogorov-Smirnov distance and Bhattacharyya coefficient. Results showed that model performance was stable over the evaluation period. Features and predictions showed statistically significant drifts; however, these drifts were not linked to changes in model performance during the time of our study.