mlsanityData to 14 Oct 2026

Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging

arXiv 2505.21698 ↗First posted 27 May 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language foundation models achieve promising performance in natural image classification, yet their direct application to medical imaging is limited by severe domain shifts, resolution mismatches, and the multi-label nature of clinical diagnosis. Training dedicated medical foundation models from scratch, however, is costly and data-intensive. Here, we propose MedBridge, a lightweight adaptation framework that opens a new direction in domain-gap mitigation by jointly combining domain alignment, resolution preservation, and multi-label reasoning via complementary VLM experts for medical image diagnosis. Specifically, MedBridge transforms pretrained VLMs into multi-view query encoders that inject a compact set of learnable query tokens into intermediate layers, enabling non-destructive domain alignment while preserving fine-grained pathological cues via multi-view high-resolution sampling. These query tokens further act as routing signals for a mixture-of-experts, dynamically integrating heterogeneous foundation models for multi-label reasoning without requiring a shared representation space. We evaluated MedBridge on five chest radiograph benchmarks in three key adaptation tasks. MedBridge demonstrates superior performance in both cross-domain generalization (out-of-distribution transfer) and in-domain specialization (same-distribution tuning) settings, yielding a significant 6-15% AUC improvement over state-of-the-art adaptation methods for multi-label thoracic disease diagnosis. Furthermore, MedBridge is model-agnostic and demonstrates broad extensibility across eight diverse VLMs (e.g., CLIP, LLaVA, Qwen-VL, MedGemma), highlighting its ability to flexibly adapt arbitrary foundation models into a powerful medical diagnostic tool. Our code will be released upon acceptance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 59 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

45 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays29 Oct
  2. 2024Few-shot Adaptation of Medical Vision-Language Models5 Sep
  3. 2024Disease-informed Adaptation of Vision-Language Models24 May
  4. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models16 Apr
  5. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework12 Mar
  6. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models15 Aug
  7. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains9 Oct
  8. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study30 Sep

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2016Layer Normalizationcited by 5,669
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  11. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
25 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  2. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  3. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  4. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  6. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  7. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  8. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  9. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  10. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  11. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  12. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  13. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 106
  14. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
  15. 2024Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Imagescited by 69
  16. 2023CLIP in Medical Imaging: A Surveycited by 40
  17. 2025MMRL: Multi-Modal Representation Learning for Vision-Language Modelscited by 39
  18. 2024MediCLIP: Adapting CLIP for Few-shot Medical Image Anomaly Detectioncited by 35
  19. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Modelcited by 31
  20. 2023Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challengecited by 22
  21. 2024Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Modelscited by 18
  22. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  23. 2023Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentationcited by 11
  24. 2023COVIDx CXR-4: An Expanded Multi-Institutional Open-Source Benchmark Dataset for Chest X-ray Image-Based Computer-Aided COVID-19 Diagnosticscited by 6
  25. 2025Vision-Language Model Selection and Reuse for Downstream Adaptationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Representation-guided in-context learning for medical image interpretation with multimodal large language models
  2. 2024Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training
  3. 2026HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts
  4. 2025Scaling Down to Scale Up: Towards Operationally-Efficient and Deployable Clinical Models via Cross-Modal Low-Rank Adaptation for Medical Vision-Language Models
  5. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning
  6. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  7. 2026MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models
  8. 2025See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis