# Deep Hypothesis Tests Detect Clinically Relevant Subgroup Shifts in Medical Images

arXiv: https://arxiv.org/abs/2303.04862 · HTML: https://mlsanity.com/paper/2303.04862/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy](https://mlsanity.com/paper/1611.04488/) (2016)
- [Distilling Model Failures as Directions in Latent Space](https://mlsanity.com/paper/2206.14754/) (2022)
- [Classification Logit Two-sample Testing by Neural Networks](https://mlsanity.com/paper/1909.11298/) (2019)
- [AutoML Two-Sample Test](https://mlsanity.com/paper/2206.08843/) (2022)
- [Identifying the key components in ResNet-50 for diabetic retinopathy grading from fundus images: a systematic investigation](https://mlsanity.com/paper/2110.14160/) (2021)

## Built on by (most cited first, 1 of 1)

- [Automatic dataset shift identification to support safe deployment of medical imaging AI](https://mlsanity.com/paper/2411.07940/) (2024)

## Abstract (arXiv)

Distribution shifts remain a fundamental problem for the safe application of machine learning systems. If undetected, they may impact the real-world performance of such systems or will at least render original performance claims invalid. In this paper, we focus on the detection of subgroup shifts, a type of distribution shift that can occur when subgroups have a different prevalence during validation compared to the deployment setting. For example, algorithms developed on data from various acquisition settings may be predominantly applied in hospitals with lower quality data acquisition, leading to an inadvertent performance drop. We formulate subgroup shift detection in the framework of statistical hypothesis testing and show that recent state-of-the-art statistical tests can be effectively applied to subgroup shift detection on medical imaging data. We provide synthetic experiments as well as extensive evaluation on clinically meaningful subgroup shifts on histopathology as well as retinal fundus images. We conclude that classifier-based subgroup shift detection tests could be a particularly useful tool for post-market surveillance of deployed ML systems.