# Reading Race: AI Recognises Patient's Racial Identity In Medical Images

arXiv: https://arxiv.org/abs/2107.10356 · HTML: https://mlsanity.com/paper/2107.10356/

## Key facts

- Cited by 156 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 53 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation](https://mlsanity.com/paper/1801.05746/) (2018)
- [Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning](https://mlsanity.com/paper/1708.09843/) (2017)
- [Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcuts](https://mlsanity.com/paper/2009.10132/) (2020)

## Built on by (most cited first, 30 of 156)

- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Fair Conformal Predictors for Applications in Medical Imaging](https://mlsanity.com/paper/2109.04392/) (2021)
- [MEDFAIR: Benchmarking Fairness for Medical Imaging](https://mlsanity.com/paper/2210.01725/) (2022)
- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [Improving the Fairness of Chest X-ray Classifiers](https://mlsanity.com/paper/2203.12609/) (2022)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation](https://mlsanity.com/paper/2312.14223/) (2023)
- [Gender Artifacts in Visual Datasets](https://mlsanity.com/paper/2206.09191/) (2022)
- [MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision](https://mlsanity.com/paper/2308.16139/) (2023)
- [Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review](https://mlsanity.com/paper/2112.12596/) (2021)
- [BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys](https://mlsanity.com/paper/2310.10765/) (2023)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging](https://mlsanity.com/paper/2402.14815/) (2024)
- [Healthsheet: Development of a Transparency Artifact for Health Datasets](https://mlsanity.com/paper/2202.13028/) (2022)
- [Algorithm Fairness in AI for Medicine and Healthcare](https://mlsanity.com/paper/2110.00603/) (2021)
- [Can AI Relate: Testing Large Language Model Response for Mental Health Support](https://mlsanity.com/paper/2405.12021/) (2024)
- [Write It Like You See It: Detectable Differences in Clinical Notes By Race Lead To Differential Model Recommendations](https://mlsanity.com/paper/2205.03931/) (2022)
- [Detecting Shortcuts in Medical Images -- A Case Study in Chest X-rays](https://mlsanity.com/paper/2211.04279/) (2022)
- [Feature robustness and sex differences in medical imaging: a case study in MRI-based Alzheimer's disease detection](https://mlsanity.com/paper/2204.01737/) (2022)
- [Position: Use Sparse Autoencoders to Discover Unknowns](https://mlsanity.com/paper/2506.23845/) (2025)
- [Synthetically Enhanced: Unveiling Synthetic Data's Potential in Medical Imaging Research](https://mlsanity.com/paper/2311.09402/) (2023)
- [Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare](https://mlsanity.com/paper/2202.01906/) (2022)
- [A Study of Demographic Bias in CNN-based Brain MR Segmentation](https://mlsanity.com/paper/2208.06613/) (2022)
- [A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis](https://mlsanity.com/paper/2405.14839/) (2024)
- [Deep Learning Discovery of Demographic Biomarkers in Echocardiography](https://mlsanity.com/paper/2207.06421/) (2022)
- [Going public: the role of public participation approaches in commercial AI labs](https://mlsanity.com/paper/2306.09871/) (2023)
- [The Role of Subgroup Separability in Group-Fair Medical Image Classification](https://mlsanity.com/paper/2307.02791/) (2023)
- [No Fair Lunch: A Causal Perspective on Dataset Bias in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2307.16526/) (2023)
- [DMLR: Data-centric Machine Learning Research -- Past, Present and Future](https://mlsanity.com/paper/2311.13028/) (2023)

## Abstract (arXiv)

Background: In medical imaging, prior studies have demonstrated disparate AI performance by race, yet there is no known correlation for race on medical imaging that would be obvious to the human expert interpreting the images. Methods: Using private and public datasets we evaluate: A) performance quantification of deep learning models to detect race from medical images, including the ability of these models to generalize to external environments and across multiple imaging modalities, B) assessment of possible confounding anatomic and phenotype population features, such as disease distribution and body habitus as predictors of race, and C) investigation into the underlying mechanism by which AI models can recognize race. Findings: Standard deep learning models can be trained to predict race from medical images with high performance across multiple imaging modalities. Our findings hold under external validation conditions, as well as when models are optimized to perform clinically motivated tasks. We demonstrate this detection is not due to trivial proxies or imaging-related surrogate covariates for race, such as underlying disease distribution. Finally, we show that performance persists over all anatomical regions and frequency spectrum of the images suggesting that mitigation efforts will be challenging and demand further study. Interpretation: We emphasize that model ability to predict self-reported race is itself not the issue of importance. However, our findings that AI can trivially predict self-reported race -- even from corrupted, cropped, and noised medical images -- in a setting where clinical experts cannot, creates an enormous risk for all model deployments in medical imaging: if an AI model secretly used its knowledge of self-reported race to misclassify all Black patients, radiologists would not be able to tell using the same data the model has access to.