# Automatic dataset shift identification to support safe deployment of medical imaging AI

arXiv: https://arxiv.org/abs/2411.07940 · HTML: https://mlsanity.com/paper/2411.07940/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [StainNet: a fast and robust stain normalization network](https://mlsanity.com/paper/2012.12535/) (2020)
- [Classification Logit Two-sample Testing by Neural Networks](https://mlsanity.com/paper/1909.11298/) (2019)
- [Test-time Adaptation with Calibration of Medical Image Classification Nets for Label Distribution Shift](https://mlsanity.com/paper/2207.00769/) (2022)
- [Deployment of Image Analysis Algorithms under Prevalence Shifts](https://mlsanity.com/paper/2303.12540/) (2023)
- [Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens](https://mlsanity.com/paper/2311.11463/) (2023)
- [Deep Hypothesis Tests Detect Clinically Relevant Subgroup Shifts in Medical Images](https://mlsanity.com/paper/2303.04862/) (2023)

## Built on by (most cited first, 5 of 5)

- [Limitations of Public Chest Radiography Datasets for Artificial Intelligence: Label Quality, Domain Shift, Bias and Evaluation Challenges](https://mlsanity.com/paper/2509.15107/) (2025)
- [Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs](https://mlsanity.com/paper/2602.03895/) (2026)
- [Label-free estimation of clinically relevant performance metrics under distribution shifts](https://mlsanity.com/paper/2507.22776/) (2025)
- [CARL-CXR: Continual Adapter-Based Routing for Task-Unknown Chest Radiograph Classification](https://mlsanity.com/paper/2602.15811/) (2026)
- [Development and evaluation of CADe systems in low-prevalence setting: The RARE25 challenge for early detection of Barrett's neoplasia](https://mlsanity.com/paper/2604.11171/) (2026)

## Abstract (arXiv)

Shifts in data distribution can substantially harm the performance of clinical AI models and lead to misdiagnosis. Hence, various methods have been developed to detect the presence of such shifts at deployment time. However, the root causes of dataset shifts are diverse, and the choice of shift mitigation strategies is highly dependent on the precise type of shift encountered at test time. As such, detecting test-time dataset shift is not sufficient: precisely identifying which type of shift has occurred is critical. In this work, we propose the first unsupervised dataset shift identification framework for imaging datasets, effectively distinguishing between prevalence shift (caused by a change in the label distribution), covariate shift (caused by a change in input characteristics) and mixed shifts (simultaneous prevalence and covariate shifts). We discuss the importance of self-supervised encoders for detecting subtle covariate shifts and propose a novel shift detector leveraging both self-supervised encoders and task model outputs for improved shift detection. We show the effectiveness of the proposed shift identification framework across three different imaging modalities (chest radiography, digital mammography, and retinal fundus images) on five types of real-world dataset shifts using five large publicly available datasets.