# Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging

arXiv: https://arxiv.org/abs/2402.14815 · HTML: https://mlsanity.com/paper/2402.14815/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 66 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging](https://mlsanity.com/paper/1909.12475/) (2019)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Change is Hard: A Closer Look at Subpopulation Shift](https://mlsanity.com/paper/2302.12254/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Towards a Visual-Language Foundation Model for Computational Pathology](https://mlsanity.com/paper/2307.12914/) (2023)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [Are Sex-based Physiological Differences the Cause of Gender Bias for Chest X-ray Diagnosis?](https://mlsanity.com/paper/2308.05129/) (2023)
- [The Limits of Fair Medical Imaging AI In The Wild](https://mlsanity.com/paper/2312.10083/) (2023)

## Built on by (most cited first, 24 of 24)

- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.03906/) (2025)
- [Beyond Diagnostic Performance: Revealing and Quantifying Ethical Risks in Pathology Foundation Models](https://mlsanity.com/paper/2502.16889/) (2025)
- [6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models](https://mlsanity.com/paper/2512.04238/) (2025)
- [BiasICL: In-Context Learning and Demographic Biases of Vision Language Models](https://mlsanity.com/paper/2503.02334/) (2025)
- [Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches](https://mlsanity.com/paper/2506.10825/) (2025)
- [FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models](https://mlsanity.com/paper/2604.04858/) (2026)
- [debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias](https://mlsanity.com/paper/2410.13146/) (2024)
- [Understanding Dataset Bias in Medical Imaging: A Case Study on Chest X-rays](https://mlsanity.com/paper/2507.07722/) (2025)
- [Fairness and Robustness of CLIP-Based Models for Chest X-rays](https://mlsanity.com/paper/2507.21291/) (2025)
- [Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context](https://mlsanity.com/paper/2509.19671/) (2025)
- [Fairness in Multi-modal Medical Diagnosis with Demonstration Selection](https://mlsanity.com/paper/2511.15986/) (2025)
- [Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification](https://mlsanity.com/paper/2512.15249/) (2025)
- [Fairness-Aware Test-Time Prompt Tuning](https://mlsanity.com/paper/2608.25707/) (2026)
- [Secure Diagnostics: Adversarial Robustness Meets Clinical Interpretability](https://mlsanity.com/paper/2504.05483/) (2025)
- [Can Vision Transformers with ResNet's Global Features Fairly Authenticate Demographic Faces?](https://mlsanity.com/paper/2506.05383/) (2025)
- [EVLF-FM: Explainable Vision Language Foundation Model for Medicine](https://mlsanity.com/paper/2509.24231/) (2025)
- [Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2512.19663/) (2025)
- [Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models](https://mlsanity.com/paper/2601.21830/) (2026)
- [PRISM: Exploring Heterogeneous Pretrained EEG Foundation Model Transfer to Clinical Differential Diagnosis](https://mlsanity.com/paper/2603.02268/) (2026)
- [Investigating Trustworthiness of Nonparametric Deep Survival Models for Alzheimer's Disease Progression Analysis](https://mlsanity.com/paper/2605.04063/) (2026)
- [CIPHER: Causal Intervention Pathways for Healthcare Equity and Robustness](https://mlsanity.com/paper/2607.02596/) (2026)
- [Hearsay: Vision-Language Medical Diagnoses Without an Image](https://mlsanity.com/paper/2607.26886/) (2026)

## Abstract (arXiv)

Advances in artificial intelligence (AI) have achieved expert-level performance in medical imaging applications. Notably, self-supervised vision-language foundation models can detect a broad spectrum of pathologies without relying on explicit training annotations. However, it is crucial to ensure that these AI models do not mirror or amplify human biases, thereby disadvantaging historically marginalized groups such as females or Black patients. The manifestation of such biases could systematically delay essential medical care for certain patient subgroups. In this study, we investigate the algorithmic fairness of state-of-the-art vision-language foundation models in chest X-ray diagnosis across five globally-sourced datasets. Our findings reveal that compared to board-certified radiologists, these foundation models consistently underdiagnose marginalized groups, with even higher rates seen in intersectional subgroups, such as Black female patients. Such demographic biases present over a wide range of pathologies and demographic attributes. Further analysis of the model embedding uncovers its significant encoding of demographic information. Deploying AI systems with these biases in medical imaging can intensify pre-existing care disparities, posing potential challenges to equitable healthcare access and raising ethical questions about their clinical application.