mlsanityData to 14 Oct 2026

Robust Fairness Vision-Language Learning for Medical Image Analysis

arXiv 2505.03153 ↗First posted 6 May 2025cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The advent of Vision-Language Models (VLMs) in medical image analysis has the potential to help process multimodal inputs and increase performance over traditional inference methods. However, when considering the domain in which these models will be implemented, fairness and robustness are important to ensure the model stays true for any patient. In this paper, we introduce a framework for ensuring robustness and fairness of VLM models. This framework modifies the loss function at training by identifying and adjusting faulty image-text pairs through a Dynamic Bad Pair Mining algorithm and also utilizing Sinkhorn distance to ensure the loss distributions of protected groups do not deviate from the total loss. Experimental testing of our framework shows up to a 8.6% improvement when looking at equity-scaled AUC.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 57 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

42 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Vision Language Models in Medicine24 Feb
  2. 2024Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images31 May
  3. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study21 Feb

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2018Computational Optimal Transportcited by 780
  5. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  6. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  7. 2021Contrastive Learning with Stronger Augmentationscited by 100
  8. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 76
  9. 2024Preserving Fairness Generalization in Deepfake Detectioncited by 70
  10. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  11. 2022Transformers in Medical Image Analysis: A Reviewcited by 41
  12. 2023Improving Fairness in Deepfake Detectioncited by 40
27 more
  1. 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Modelscited by 32
  2. 2023Harvard Glaucoma Fairness: A Retinal Nerve Disease Dataset for Fairness Learning and Fair Identity Normalizationcited by 31
  3. 2021Robust Attentive Deep Neural Network for Exposing GAN-generated Facescited by 28
  4. 2024AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmarkcited by 28
  5. 2020Learning by Minimizing the Sum of Ranked Rangecited by 27
  6. 2024Identifying Implicit Social Biases in Vision-Language Modelscited by 27
  7. 2022Rank-based Decomposable Losses in Machine Learning: A Surveycited by 26
  8. 2021Sum of Ranked Range Loss for Supervised Learningcited by 23
  9. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  10. 2024Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Imagescited by 20
  11. 2024Robust Light-Weight Facial Affective Behavior Recognition with CLIPcited by 16
  12. 2022Distributionally Robust Survival Analysis: A Novel Fairness Loss Without Demographicscited by 15
  13. 2024UU-Mamba: Uncertainty-aware U-Mamba for Cardiac Image Segmentationcited by 15
  14. 2024EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysiscited by 15
  15. 2024Fairness in Survival Analysis with Distributionally Robust Optimizationcited by 14
  16. 2023Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approachcited by 13
  17. 2024Robust COVID-19 Detection in CT Images with CLIPcited by 12
  18. 2024Robustly Optimized Deep Feature Decoupling Network for Fatty Liver Diseases Detectioncited by 11
  19. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  20. 2022PseudoProp: Robust Pseudo-Label Generation for Semi-Supervised Object Detection in Autonomous Driving Systemscited by 10
  21. 2023UMedNeRF: Uncertainty-aware Single View Volumetric Rendering for Medical Neural Radiance Fieldscited by 10
  22. 2024Uncertainty-Aware Explainable Recommendation with Large Language Modelscited by 10
  23. 2025Diffusion-empowered AutoPrompt MedSAMcited by 10
  24. 2024FACMIC: Federated Adaptative CLIP Model for Medical Image Classificationcited by 7
  25. 2024An Explainable Non-local Network for COVID-19 Diagnosiscited by 5
  26. 2024Rethinking Positive Pairs in Contrastive Learningcited by 5
  27. 2025The Reliability of LLMs for Medical Diagnosis: An Examination of Consistency, Manipulation, and Contextual Awarenesscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025MedChat: A Multi-Agent Framework for Multimodal Diagnosis with Large Language Modelscited by 5
  2. 2026Robust Multi-Source Covid-19 Detection in CT Imagescited by 0
  3. 2026Robust Fair Disease Diagnosis in CT Imagescited by 0
  4. 2026FairEnc: A Fair Vision-Language Model with Fair Vision and Text Encoders for Glaucoma Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
  2. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning
  3. 2025Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models
  4. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  5. 2025Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
  6. 2024Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training
  7. 2025Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models
  8. 2026Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction