# Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging

arXiv: https://arxiv.org/abs/1909.12475 · HTML: https://mlsanity.com/paper/1909.12475/

## Key facts

- Cited by 216 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 30 of 216)

- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations](https://mlsanity.com/paper/2203.01517/) (2022)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [On Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2210.11369/) (2022)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [The Fallacy of AI Functionality](https://mlsanity.com/paper/2206.09511/) (2022)
- [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022)
- [Simple and Fast Group Robustness by Automatic Feature Reweighting](https://mlsanity.com/paper/2306.11074/) (2023)
- [The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models](https://mlsanity.com/paper/2107.00758/) (2021)
- [Contrastive Adapters for Foundation Model Group Robustness](https://mlsanity.com/paper/2207.07180/) (2022)
- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Diversify and Disambiguate: Learning From Underspecified Data](https://mlsanity.com/paper/2202.03418/) (2022)
- [Mandoline: Model Evaluation under Distribution Shift](https://mlsanity.com/paper/2107.00643/) (2021)
- [Towards Last-layer Retraining for Group Robustness with Fewer Annotations](https://mlsanity.com/paper/2309.08534/) (2023)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation](https://mlsanity.com/paper/2312.14223/) (2023)
- [Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning](https://mlsanity.com/paper/2204.07596/) (2022)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [An Empirical Framework for Domain Generalization in Clinical Settings](https://mlsanity.com/paper/2103.11163/) (2021)
- [Avoiding spurious correlations via logit correction](https://mlsanity.com/paper/2212.01433/) (2022)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization](https://mlsanity.com/paper/2107.12580/) (2021)
- [Simple and near-optimal algorithms for hidden stratification and multi-group learning](https://mlsanity.com/paper/2112.12181/) (2021)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [FACTS: First Amplify Correlations and Then Slice to Discover Bias](https://mlsanity.com/paper/2309.17430/) (2023)
- [BARACK: Partially Supervised Group Robustness With Guarantees](https://mlsanity.com/paper/2201.00072/) (2021)

## Abstract (arXiv)

Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model still consistently misses a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring and describing hidden stratification effects, and characterize these effects on multiple medical imaging datasets. We find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we explore the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.