# CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI

arXiv: https://arxiv.org/abs/2202.02833 · HTML: https://mlsanity.com/paper/2202.02833/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 6 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)
- [A Benchmark of Medical Out of Distribution Detection](https://mlsanity.com/paper/2007.04250/) (2020)
- [Monitoring and explainability of models in production](https://mlsanity.com/paper/2007.06299/) (2020)
- [Automatically detecting data drift in machine learning classifiers](https://mlsanity.com/paper/2111.05672/) (2021)
- [A Less Biased Evaluation of Out-of-distribution Sample Detectors](https://mlsanity.com/paper/1809.04729/) (2018)
- [Integrating AI into Radiology workflow: Levels of research, production, and feedback maturity](https://mlsanity.com/paper/1910.06424/) (2019)
- [Concept Drift Detection and Adaptation with Weak Supervision on Streaming Unlabeled Data](https://mlsanity.com/paper/1910.01064/) (2019)

## Built on by (most cited first, 7 of 7)

- [Statistically Valid Post-Deployment Monitoring Should Be Standard for AI-Based Digital Health](https://mlsanity.com/paper/2506.05701/) (2025)
- [Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach](https://mlsanity.com/paper/2410.21346/) (2024)
- [MLHOps: Machine Learning for Healthcare Operations](https://mlsanity.com/paper/2305.02474/) (2023)
- [Out-of-Distribution Detection and Data Drift Monitoring using Statistical Process Control](https://mlsanity.com/paper/2402.08088/) (2024)
- [A quality assurance framework for real-time monitoring of deep learning segmentation models in radiotherapy](https://mlsanity.com/paper/2305.11715/) (2023)
- [Domain Shift Analysis in Chest Radiographs Classification in a Veterans Healthcare Administration Population](https://mlsanity.com/paper/2407.21149/) (2024)
- [Trimming the Risk: Towards Reliable Continuous Training for Deep Learning Inspection Systems](https://mlsanity.com/paper/2409.09108/) (2024)

## Abstract (arXiv)

Clinical Artificial lntelligence (AI) applications are rapidly expanding worldwide, and have the potential to impact to all areas of medical practice. Medical imaging applications constitute a vast majority of approved clinical AI applications. Though healthcare systems are eager to adopt AI solutions a fundamental question remains: textitwhat happens after the AI model goes into production? We use the CheXpert and PadChest public datasets to build and test a medical imaging AI drift monitoring workflow to track data and model drift without contemporaneous ground truth. We simulate drift in multiple experiments to compare model performance with our novel multi-modal drift metric, which uses DICOM metadata, image appearance representation from a variational autoencoder (VAE), and model output probabilities as input. Through experimentation, we demonstrate a strong proxy for ground truth performance using unsupervised distributional shifts in relevant metadata, predicted probabilities, and VAE latent representation. Our key contributions include (1) proof-of-concept for medical imaging drift detection that includes the use of VAE and domain specific statistical methods, (2) a multi-modal methodology to measure and unify drift metrics, (3) new insights into the challenges and solutions to observe deployed medical imaging AI, and (4) creation of open-source tools that enable others to easily run their own workflows and scenarios. This work has important implications. It addresses the concerning translation gap found in continuous medical imaging AI model monitoring common in dynamic healthcare environments.