# Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications

arXiv: https://arxiv.org/abs/2407.16953 · HTML: https://mlsanity.com/paper/2407.16953/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Understanding metric-related pitfalls in image analysis validation](https://mlsanity.com/paper/2302.01790/) (2023)
- [Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentation](https://mlsanity.com/paper/2106.12387/) (2021)
- [Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers](https://mlsanity.com/paper/2203.04913/) (2022)
- [Diagnosing failures of fairness transfer across distribution shift in real-world medical settings](https://mlsanity.com/paper/2202.01034/) (2022)
- [The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default](https://mlsanity.com/paper/2302.02404/) (2023)
- [Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth](https://mlsanity.com/paper/1702.03407/) (2017)
- [Estimating and Improving Fairness with Adversarial Learning](https://mlsanity.com/paper/2103.04243/) (2021)
- [Epistemic Uncertainty-Weighted Loss for Visual Bias Mitigation](https://mlsanity.com/paper/2204.09389/) (2022)
- [FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling](https://mlsanity.com/paper/2303.00586/) (2023)
- [Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentation](https://mlsanity.com/paper/2302.05432/) (2023)
- [Towards unraveling calibration biases in medical image analysis](https://mlsanity.com/paper/2305.05101/) (2023)
- [Are demographically invariant models and representations in medical imaging fair?](https://mlsanity.com/paper/2305.01397/) (2023)
- [Measuring Diversity of Artificial Intelligence Conferences](https://mlsanity.com/paper/2001.07038/) (2020)
- [Two-step adversarial debiasing with partial learning -- medical image case-studies](https://mlsanity.com/paper/2111.08711/) (2021)

## Built on by (most cited first, 2 of 2)

- [BM-CL: Bias Mitigation through the lens of Continual Learning](https://mlsanity.com/paper/2509.01730/) (2025)
- [Sex-based Bias Inherent in the Dice Similarity Coefficient: A Model Independent Analysis for Multiple Anatomical Structures](https://mlsanity.com/paper/2509.19778/) (2025)

## Abstract (arXiv)

Recently, the research community of computerized medical imaging has started to discuss and address potential fairness issues that may emerge when developing and deploying AI systems for medical image analysis. This chapter covers some of the pressing challenges encountered when doing research in this area, and it is intended to raise questions and provide food for thought for those aiming to enter this research field. The chapter first discusses various sources of bias, including data collection, model training, and clinical deployment, and their impact on the fairness of machine learning algorithms in medical image computing. We then turn to discussing open challenges that we believe require attention from researchers and practitioners, as well as potential pitfalls of naive application of common methods in the field. We cover a variety of topics including the impact of biased metrics when auditing for fairness, the leveling down effect, task difficulty variations among subgroups, discovering biases in unseen populations, and explaining biases beyond standard demographic attributes.