# Deployment of Image Analysis Algorithms under Prevalence Shifts

arXiv: https://arxiv.org/abs/2303.12540 · HTML: https://mlsanity.com/paper/2303.12540/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 46 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [Tutorial: Safe and Reliable Machine Learning](https://mlsanity.com/paper/1904.07204/) (2019)
- [MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs](https://mlsanity.com/paper/1712.06957/) (2017)
- [Test-time Adaptation with Calibration of Medical Image Classification Nets for Label Distribution Shift](https://mlsanity.com/paper/2207.00769/) (2022)
- [Analysis and Comparison of Classification Metrics](https://mlsanity.com/paper/2209.05355/) (2022)
- [Preventing dataset shift from breaking machine-learning biomarkers](https://mlsanity.com/paper/2107.09947/) (2021)

## Built on by (most cited first, 4 of 4)

- [Automatic dataset shift identification to support safe deployment of medical imaging AI](https://mlsanity.com/paper/2411.07940/) (2024)
- [Application-driven Validation of Posteriors in Inverse Problems](https://mlsanity.com/paper/2309.09764/) (2023)
- [On the Stability of a non-hyperbolic nonlinear map with non-bounded set of non-isolated fixed points with applications to Machine Learning](https://mlsanity.com/paper/2401.03051/) (2024)
- [A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift](https://mlsanity.com/paper/2605.20956/) (2026)

## Abstract (arXiv)

Domain gaps are among the most relevant roadblocks in the clinical translation of machine learning (ML)-based solutions for medical image analysis. While current research focuses on new training paradigms and network architectures, little attention is given to the specific effect of prevalence shifts on an algorithm deployed in practice. Such discrepancies between class frequencies in the data used for a method's development/validation and that in its deployment environment(s) are of great importance, for example in the context of artificial intelligence (AI) democratization, as disease prevalences may vary widely across time and location. Our contribution is twofold. First, we empirically demonstrate the potentially severe consequences of missing prevalence handling by analyzing (i) the extent of miscalibration, (ii) the deviation of the decision threshold from the optimum, and (iii) the ability of validation metrics to reflect neural network performance on the deployment population as a function of the discrepancy between development and deployment prevalence. Second, we propose a workflow for prevalence-aware image classification that uses estimated deployment prevalences to adjust a trained classifier to a new environment, without requiring additional annotated deployment data. Comprehensive experiments based on a diverse set of 30 medical classification tasks showcase the benefit of the proposed workflow in generating better classifier decisions and more reliable performance estimates compared to current practice.