# Expert-Driven Monitoring of Operational ML Models

arXiv: https://arxiv.org/abs/2401.11993 · HTML: https://mlsanity.com/paper/2401.11993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 27 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 9 of 9)

- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Characterizing Concept Drift](https://mlsanity.com/paper/1511.03816/) (2015)
- [Mandoline: Model Evaluation under Distribution Shift](https://mlsanity.com/paper/2107.00643/) (2021)
- [On the Reliable Detection of Concept Drift from Streaming Unlabeled Data](https://mlsanity.com/paper/1704.00023/) (2017)
- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [Towards Observability for Production Machine Learning Pipelines](https://mlsanity.com/paper/2108.13557/) (2021)
- [Understanding Unfairness in Fraud Detection through Model and Data Bias Interactions](https://mlsanity.com/paper/2207.06273/) (2022)
- [Identifying the Context Shift between Test Benchmarks and Production Data](https://mlsanity.com/paper/2207.01059/) (2022)

## Abstract (arXiv)

We propose Expert Monitoring, an approach that leverages domain expertise to enhance the detection and mitigation of concept drift in machine learning (ML) models. Our approach supports practitioners by consolidating domain expertise related to concept drift-inducing events, making this expertise accessible to on-call personnel, and enabling automatic adaptability with expert oversight.