# FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models

arXiv: https://arxiv.org/abs/2407.00983 · HTML: https://mlsanity.com/paper/2407.00983/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 84 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Fairness in Machine Learning: A Survey](https://mlsanity.com/paper/2010.04053/) (2020)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Generative models improve fairness of medical classifiers under distribution shifts](https://mlsanity.com/paper/2304.09218/) (2023)
- [SegVol: Universal and Interactive Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2311.13385/) (2023)
- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representations](https://mlsanity.com/paper/2007.07423/) (2020)

## Built on by (most cited first, 30 of 47)

- [A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges](https://mlsanity.com/paper/2501.02189/) (2025)
- [A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcare](https://mlsanity.com/paper/2502.15871/) (2025)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [EchoApex: A General-Purpose Vision Foundation Model for Echocardiography](https://mlsanity.com/paper/2410.11092/) (2024)
- [Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification](https://mlsanity.com/paper/2506.23298/) (2025)
- [Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.03906/) (2025)
- [FairREAD: Re-fusing Demographic Attributes after Disentanglement for Fair Medical Image Classification](https://mlsanity.com/paper/2412.16373/) (2024)
- [Bias and Generalizability of Foundation Models across Datasets in Breast Mammography](https://mlsanity.com/paper/2505.10579/) (2025)
- [Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging](https://mlsanity.com/paper/2506.18434/) (2025)
- [Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives](https://mlsanity.com/paper/2502.16841/) (2025)
- [Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench](https://mlsanity.com/paper/2601.02737/) (2026)
- [Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language](https://mlsanity.com/paper/2603.02790/) (2026)
- [SCAN: Structured Capability Assessment and Navigation for LLMs](https://mlsanity.com/paper/2505.06698/) (2025)
- [See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis](https://mlsanity.com/paper/2506.18140/) (2025)
- [AdFair-CLIP: Adversarial Fair Contrastive Language-Image Pre-training for Chest X-rays](https://mlsanity.com/paper/2506.23467/) (2025)
- [Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classification](https://mlsanity.com/paper/2501.19086/) (2025)
- [Distribution-aware Fairness Learning in Medical Image Segmentation From A Control-Theoretic Perspective](https://mlsanity.com/paper/2502.00619/) (2025)
- [FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classification](https://mlsanity.com/paper/2503.18981/) (2025)
- [VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models](https://mlsanity.com/paper/2507.00052/) (2025)
- [CardioBench: Do Echocardiography Foundation Models Generalize Beyond the Lab?](https://mlsanity.com/paper/2510.00520/) (2025)
- [Surgeons Are Indian Males and Speech Therapists Are White Females: Auditing Biases in Vision-Language Models for Healthcare Professionals](https://mlsanity.com/paper/2510.06280/) (2025)
- [On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detection](https://mlsanity.com/paper/2511.07700/) (2025)
- [OxEnsemble: Fair Ensembles for Low-Data Classification](https://mlsanity.com/paper/2512.09665/) (2025)
- [A Hierarchical Benchmark of Foundation Models for Dermatology](https://mlsanity.com/paper/2601.12382/) (2026)
- [Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs](https://mlsanity.com/paper/2602.03895/) (2026)
- [DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents](https://mlsanity.com/paper/2603.00777/) (2026)
- [FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants](https://mlsanity.com/paper/2603.26008/) (2026)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [Expert-Like Reparameterization of Heterogeneous Pyramid Receptive Fields in Efficient CNNs for Fair Medical Image Classification](https://mlsanity.com/paper/2505.13039/) (2025)
- [One Size Fits None: Rethinking Fairness in Medical AI](https://mlsanity.com/paper/2506.14400/) (2025)

## Abstract (arXiv)

The advent of foundation models (FMs) in healthcare offers unprecedented opportunities to enhance medical diagnostics through automated classification and segmentation tasks. However, these models also raise significant concerns about their fairness, especially when applied to diverse and underrepresented populations in healthcare applications. Currently, there is a lack of comprehensive benchmarks, standardized pipelines, and easily adaptable libraries to evaluate and understand the fairness performance of FMs in medical imaging, leading to considerable challenges in formulating and implementing solutions that ensure equitable outcomes across diverse patient populations. To fill this gap, we introduce FairMedFM, a fairness benchmark for FM research in medical imaging.FairMedFM integrates with 17 popular medical imaging datasets, encompassing different modalities, dimensionalities, and sensitive attributes. It explores 20 widely used FMs, with various usages such as zero-shot learning, linear probing, parameter-efficient fine-tuning, and prompting in various downstream tasks -- classification and segmentation. Our exhaustive analysis evaluates the fairness performance over different evaluation metrics from multiple perspectives, revealing the existence of bias, varied utility-fairness trade-offs on different FMs, consistent disparities on the same datasets regardless FMs, and limited effectiveness of existing unfairness mitigation methods. Checkout FairMedFM's project page and open-sourced codebase, which supports extendible functionalities and applications as well as inclusive for studies on FMs in medical imaging over the long term.