mlsanityData to 7 Oct 2026

Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging

arXiv 1909.12475 ↗First posted 27 Sep 2019cs.LG stat.MLcited by 215

Not on the map. Why?

Abstract (arXiv)

Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model still consistently misses a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring and describing hidden stratification effects, and characterize these effects on multiple medical imaging datasets. We find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we explore the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 37; 2023: 39; 2024: 28; 2025: 31; 2026: 54.

215 papers.

Sort:

By area:

  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  2. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  3. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 184
  4. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  5. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  6. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  7. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  8. 2022The Fallacy of AI Functionalitycited by 88
  9. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  10. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  11. 2021The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Modelscited by 63
  12. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
38 more
  1. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  2. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
  3. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  4. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
  5. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  6. 2023SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysiscited by 44
  7. 2023Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generationcited by 41
  8. 2022Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learningcited by 40
  9. 2021How I failed machine learning in medical imaging -- shortcomings and recommendationscited by 37
  10. 2021An Empirical Framework for Domain Generalization in Clinical Settingscited by 32
  11. 2022Avoiding spurious correlations via logit correctioncited by 32
  12. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  13. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  14. 2021Simple and near-optimal algorithms for hidden stratification and multi-group learningcited by 26
  15. 2022Risk of Bias in Chest Radiography Deep Learning Foundation Modelscited by 26
  16. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  17. 2023FACTS: First Amplify Correlations and Then Slice to Discover Biascited by 26
  18. 2021BARACK: Partially Supervised Group Robustness With Guaranteescited by 25
  19. 2024Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imagingcited by 24
  20. 2023WBCAtt: A White Blood Cell Dataset Annotated with Detailed Morphological Attributescited by 22
  21. 2022Detecting Shortcuts in Medical Images -- A Case Study in Chest X-rayscited by 20
  22. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  23. 2022Symphony: Composing Interactive Interfaces for Machine Learningcited by 18
  24. 2022Chroma-VAE: Mitigating Shortcut Learning with Generative Classifierscited by 17
  25. 2021Preventing dataset shift from breaking machine-learning biomarkerscited by 14
  26. 2022Pseudo Bias-Balanced Learning for Debiased Chest X-ray Classificationcited by 14
  27. 2022Distributionally Robust Optimization via Ball Oracle Accelerationcited by 14
  28. 2022DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systemscited by 14
  29. 2023Distributionally Robust Optimization with Probabilistic Groupcited by 14
  30. 2024Slicing Through Bias: Explaining Performance Gaps in Medical Image Analysis using Slice Discovery Methodscited by 14
  31. 2022Zipfian environments for Reinforcement Learningcited by 13
  32. 2021A comparison of approaches to improve worst-case predictive model performance over patient subpopulationscited by 12
  33. 2022Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfallscited by 12
  34. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  35. 2023On (assessing) the fairness of risk score modelscited by 12
  36. 2022A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentationcited by 11
  37. 2022Modeling the Machine Learning Multiversecited by 11
  38. 2023Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamicscited by 11

50 most cited of 215 All 215 (JSON)