# Risk of Training Diagnostic Algorithms on Data with Demographic Bias

arXiv: https://arxiv.org/abs/2005.10050 · HTML: https://mlsanity.com/paper/2005.10050/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 23 of 23)

- [Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset](https://mlsanity.com/paper/2104.09957/) (2021)
- [Generative models improve fairness of medical classifiers under distribution shifts](https://mlsanity.com/paper/2304.09218/) (2023)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Addressing Fairness Issues in Deep Learning-Based Medical Image Analysis: A Systematic Review](https://mlsanity.com/paper/2209.13177/) (2022)
- [Estimating and Improving Fairness with Adversarial Learning](https://mlsanity.com/paper/2103.04243/) (2021)
- [FairAdaBN: Mitigating unfairness with adaptive batch normalization and its application to dermatological disease classification](https://mlsanity.com/paper/2303.08325/) (2023)
- [A Study of Demographic Bias in CNN-based Brain MR Segmentation](https://mlsanity.com/paper/2208.06613/) (2022)
- [Preventing dataset shift from breaking machine-learning biomarkers](https://mlsanity.com/paper/2107.09947/) (2021)
- [A systematic study of race and sex bias in CNN-based cardiac MR segmentation](https://mlsanity.com/paper/2209.01627/) (2022)
- [A Framework for Interpretability in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2310.01685/) (2023)
- [In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review](https://mlsanity.com/paper/2501.10727/) (2025)
- [Augmenting Chest X-ray Datasets with Non-Expert Annotations](https://mlsanity.com/paper/2309.02244/) (2023)
- [Copycats: the many lives of a publicly available medical imaging dataset](https://mlsanity.com/paper/2402.06353/) (2024)
- [ENHANCE (ENriching Health data by ANnotations of Crowd and Experts): A case study for skin lesion classification](https://mlsanity.com/paper/2107.12734/) (2021)
- [DeepRepViz: Identifying Confounders in Deep Learning Model Predictions](https://mlsanity.com/paper/2309.15551/) (2023)
- [Dataset Distribution Impacts Model Fairness: Single vs. Multi-Task Learning](https://mlsanity.com/paper/2407.17543/) (2024)
- [Understanding-informed Bias Mitigation for Fair CMR Segmentation](https://mlsanity.com/paper/2503.17089/) (2025)
- [Bayesian uncertainty-weighted loss for improved generalisability on polyp segmentation task](https://mlsanity.com/paper/2309.06807/) (2023)
- [An Investigation Into Race Bias in Random Forest Models Based on Breast DCE-MRI Derived Radiomics Features](https://mlsanity.com/paper/2309.17197/) (2023)
- [On the Readiness of Scientific Data for a Fair and Transparent Use in Machine Learning](https://mlsanity.com/paper/2401.10304/) (2024)
- [NeuroLIP: Interpretable and Fair Cross-Modal Alignment of fMRI and Phenotypic Text](https://mlsanity.com/paper/2503.21964/) (2025)
- [Robustness and sex differences in skin cancer detection: logistic regression vs CNNs](https://mlsanity.com/paper/2504.11415/) (2025)
- [Effect of Demographic Bias on Skin Lesion Classification](https://mlsanity.com/paper/2606.03214/) (2026)

## Abstract (arXiv)

One of the critical challenges in machine learning applications is to have fair predictions. There are numerous recent examples in various domains that convincingly show that algorithms trained with biased datasets can easily lead to erroneous or discriminatory conclusions. This is even more crucial in clinical applications where the predictive algorithms are designed mainly based on a limited or given set of medical images and demographic variables such as age, sex and race are not taken into account. In this work, we conduct a survey of the MICCAI 2018 proceedings to investigate the common practice in medical image analysis applications. Surprisingly, we found that papers focusing on diagnosis rarely describe the demographics of the datasets used, and the diagnosis is purely based on images. In order to highlight the importance of considering the demographics in diagnosis tasks, we used a publicly available dataset of skin lesions. We then demonstrate that a classifier with an overall area under the curve (AUC) of 0.83 has variable performance between 0.76 and 0.91 on subgroups based on age and sex, even though the training set was relatively balanced. Moreover, we show that it is possible to learn unbiased features by explicitly using demographic variables in an adversarial training setup, which leads to balanced scores per subgroups. Finally, we discuss the implications of these results and provide recommendations for further research.