# Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models

arXiv: https://arxiv.org/abs/2111.13657 · HTML: https://mlsanity.com/paper/2111.13657/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment](https://mlsanity.com/paper/1903.01689/) (2019)
- [Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud](https://mlsanity.com/paper/2109.03285/) (2021)
- [Optimal Quantile Approximation in Streams](https://mlsanity.com/paper/1603.05346/) (2016)
- [Relative Error Streaming Quantiles](https://mlsanity.com/paper/2004.01668/) (2020)

## Built on by (most cited first, 16 of 16)

- [DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems](https://mlsanity.com/paper/2211.05764/) (2022)
- [A Human-Centric Take on Model Monitoring](https://mlsanity.com/paper/2206.02868/) (2022)
- [Model-driven Engineering for Machine Learning Components: A Systematic Literature Review](https://mlsanity.com/paper/2311.00284/) (2023)
- [Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation](https://mlsanity.com/paper/2506.17442/) (2025)
- [The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research](https://mlsanity.com/paper/2310.07882/) (2023)
- [Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance](https://mlsanity.com/paper/2408.13648/) (2024)
- [Can Attribution Predict Risk? From Multi-View Attribution to Planning Risk Signals in End-to-End Autonomous Driving](https://mlsanity.com/paper/2605.06264/) (2026)
- [Characterizing and Measuring Linguistic Dataset Drift](https://mlsanity.com/paper/2305.17127/) (2023)
- [Towards Cross-Modal Error Detection with Tables and Images](https://mlsanity.com/paper/2510.12383/) (2025)
- [AlerTiger: Deep Learning for AI Model Health Monitoring at LinkedIn](https://mlsanity.com/paper/2306.01977/) (2023)
- [A Framework for Monitoring and Retraining Language Models in Real-World Applications](https://mlsanity.com/paper/2311.09930/) (2023)
- [Root Causing Prediction Anomalies Using Explainable AI](https://mlsanity.com/paper/2403.02439/) (2024)
- [Reliably Detecting Model Failures in Deployment Without Labels](https://mlsanity.com/paper/2506.05047/) (2025)
- [Shift Detection and Adaptation for Network Intrusion Detection](https://mlsanity.com/paper/2508.15100/) (2025)
- [Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild](https://mlsanity.com/paper/2604.16304/) (2026)
- [FMMO: Detecting the Divergence Between Local Attribution and Global Drift](https://mlsanity.com/paper/2609.06173/) (2026)

## Abstract (arXiv)

With the increasing adoption of machine learning (ML) models and systems in high-stakes settings across different industries, guaranteeing a model's performance after deployment has become crucial. Monitoring models in production is a critical aspect of ensuring their continued performance and reliability. We present Amazon SageMaker Model Monitor, a fully managed service that continuously monitors the quality of machine learning models hosted on Amazon SageMaker. Our system automatically detects data, concept, bias, and feature attribution drift in models in real-time and provides alerts so that model owners can take corrective actions and thereby maintain high quality models. We describe the key requirements obtained from customers, system design and architecture, and methodology for detecting different types of drift. Further, we provide quantitative evaluations followed by use cases, insights, and lessons learned from more than two years of production deployment.