mlsanityData to 7 Oct 2026

FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models

arXiv 2407.00983 ↗First posted 1 Jul 2024cs.CVcited by 47

Not on the map. Why?

Abstract (arXiv)

The advent of foundation models (FMs) in healthcare offers unprecedented opportunities to enhance medical diagnostics through automated classification and segmentation tasks. However, these models also raise significant concerns about their fairness, especially when applied to diverse and underrepresented populations in healthcare applications. Currently, there is a lack of comprehensive benchmarks, standardized pipelines, and easily adaptable libraries to evaluate and understand the fairness performance of FMs in medical imaging, leading to considerable challenges in formulating and implementing solutions that ensure equitable outcomes across diverse patient populations. To fill this gap, we introduce FairMedFM, a fairness benchmark for FM research in medical imaging.FairMedFM integrates with 17 popular medical imaging datasets, encompassing different modalities, dimensionalities, and sensitive attributes. It explores 20 widely used FMs, with various usages such as zero-shot learning, linear probing, parameter-efficient fine-tuning, and prompting in various downstream tasks -- classification and segmentation. Our exhaustive analysis evaluates the fairness performance over different evaluation metrics from multiple perspectives, revealing the existence of bias, varied utility-fairness trade-offs on different FMs, consistent disparities on the same datasets regardless FMs, and limited effectiveness of existing unfairness mitigation methods. Checkout FairMedFM's project page and open-sourced codebase, which supports extendible functionalities and applications as well as inclusive for studies on FMs in medical imaging over the long term.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 49 of its 84 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

49 of 84 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  11. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
37 more
  1. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  2. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  3. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  4. 2020Fairness in Machine Learning: A Surveycited by 369
  5. 2018Learning Adversarially Fair and Transferable Representationscited by 353
  6. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  7. 2018AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Biascited by 319
  8. 2023Customized Segment Anything Model for Medical Image Segmentationcited by 277
  9. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
  10. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  11. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  12. 2023The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CTcited by 101
  13. 2023A Survey on Fairness in Large Language Modelscited by 91
  14. 2023Generative models improve fairness of medical classifiers under distribution shiftscited by 90
  15. 2023SegVol: Universal and Interactive Volumetric Medical Image Segmentationcited by 86
  16. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  17. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  18. 2020Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representationscited by 69
  19. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  20. 2023A Closer Look at the Few-Shot Adaptation of Large Vision-Language Modelscited by 67
  21. 2022MEDFAIR: Benchmarking Fairness for Medical Imagingcited by 66
  22. 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
  23. 2020MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Modelscited by 61
  24. 2023LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matchingcited by 60
  25. 2021Fairness in Cardiac MR Image Analysis: An Investigation of Bias Due to Data Imbalance in Deep Learning Based Segmentationcited by 57
  26. 2022Improving the Fairness of Chest X-ray Classifierscited by 52
  27. 2023Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentationcited by 51
  28. 2023TinySAM: Pushing the Envelope for Efficient Segment Anything Modelcited by 33
  29. 2021RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHRcited by 32
  30. 2023Harvard Glaucoma Fairness: A Retinal Nerve Disease Dataset for Fairness Learning and Fair Identity Normalizationcited by 31
  31. 2023Lumbar spine segmentation in MR images: a dataset and a public benchmarkcited by 30
  32. 2023FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysiscited by 23
  33. 2023FairAdaBN: Mitigating unfairness with adaptive batch normalization and its application to dermatological disease classificationcited by 18
  34. 2024FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Imagescited by 14
  35. 2023Unveiling Fairness Biases in Deep Learning-Based Brain MRI Reconstructioncited by 5
  36. 2023Analysing race and sex bias in brain age predictioncited by 4
  37. 2024APPLE: Adversarial Privacy-aware Perturbations on Latent Embedding for Unfairness Mitigationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 25; 2026: 19.

47 papers.

Sort:

By area:

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  2. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  3. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 16
  4. 2024EchoApex: A General-Purpose Vision Foundation Model for Echocardiographycited by 11
  5. 2025Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classificationcited by 5
  6. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
  7. 2024FairREAD: Re-fusing Demographic Attributes after Disentanglement for Fair Medical Image Classificationcited by 3
  8. 2025Fair Foundation Models for Medical Image Analysis: Challenges and Perspectivescited by 3
  9. 2025Bias and Generalizability of Foundation Models across Datasets in Breast Mammographycited by 3
  10. 2025Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imagingcited by 3
  11. 2026Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Benchcited by 3
  12. 2026Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Languagecited by 3
35 more
  1. 2025See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosiscited by 2
  2. 2025Fairness Analysis of CLIP-Based Foundation Models for X-Ray Image Classificationcited by 1
  3. 2025Distribution-aware Fairness Learning in Medical Image Segmentation From A Control-Theoretic Perspectivecited by 1
  4. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classificationcited by 1
  5. 2025SCAN: Structured Capability Assessment and Navigation for LLMscited by 1
  6. 2025AdFair-CLIP: Adversarial Fair Contrastive Language-Image Pre-training for Chest X-rayscited by 1
  7. 2025VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Modelscited by 1
  8. 2025Surgeons Are Indian Males and Speech Therapists Are White Females: Auditing Biases in Vision-Language Models for Healthcare Professionalscited by 1
  9. 2025On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detectioncited by 1
  10. 2025OxEnsemble: Fair Ensembles for Low-Data Classificationcited by 1
  11. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMscited by 1
  12. 2026DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agentscited by 1
  13. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0
  14. 2025Expert-Like Reparameterization of Heterogeneous Pyramid Receptive Fields in Efficient CNNs for Fair Medical Image Classificationcited by 0
  15. 2025One Size Fits None: Rethinking Fairness in Medical AIcited by 0
  16. 2025Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Modelscited by 0
  17. 2025CardioBench: Do Echocardiography Foundation Models Generalize Beyond the Lab?cited by 0
  18. 2025MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulationcited by 0
  19. 2025Benchmarking Deep Learning Models for Laryngeal Cancer Staging Using the LaryngealCT Datasetcited by 0
  20. 2025EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoningcited by 0
  21. 2026EvalBlocks: A Modular Pipeline for Rapidly Evaluating Foundation Models in Medical Imagingcited by 0
  22. 2026A pipeline for enabling path-specific causal fairness in observational health datacited by 0
  23. 2026A Hierarchical Benchmark of Foundation Models for Dermatologycited by 0
  24. 2026Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Modelscited by 0
  25. 2026Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offscited by 0
  26. 2026FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistantscited by 0
  27. 2026Knowledge Transfer Scaling Laws for 3D Medical Imagingcited by 0
  28. 2026Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-texorpdfstringδdelta Alignmentcited by 0
  29. 2026False Confidence: Automated Labels Confound Fairness Audits in Cervical Spine Segmentationcited by 0
  30. 2026RED-Sphere: Hyperspherical Residual Edge Debiasing for Cross-Population Fundus Disease Domain Generalizationcited by 0
  31. 2026Position: Fairness Failure in Generative Models is an Evaluation Problemcited by 0
  32. 2026Subgroup performance analysis of adaptation strategies for chest X-ray foundation modelscited by 0
  33. 2026CRS-Bench: A Reference-Relative Reliability Benchmark for Medical Image Encoderscited by 0
  34. 2026Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screeningcited by 0
  35. 2026A Data-Interventional Framework for Auditing Privacy and Fairness in Generative Medical Imagingcited by 0