mlsanityData to 7 Oct 2026

Disease-informed Adaptation of Vision-Language Models

arXiv 2405.15728 ↗First posted 24 May 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

In medical image analysis, the expertise scarcity and the high cost of data annotation limits the development of large artificial intelligence models. This paper investigates the potential of transfer learning with pre-trained vision-language models (VLMs) in this domain. Currently, VLMs still struggle to transfer to the underrepresented diseases with minimal presence and new diseases entirely absent from the pretraining dataset. We argue that effective adaptation of VLMs hinges on the nuanced representation learning of disease concepts. By capitalizing on the joint visual-linguistic capabilities of VLMs, we introduce disease-informed contextual prompting in a novel disease prototype learning framework. This approach enables VLMs to grasp the concepts of new disease effectively and efficiently, even with limited data. Extensive experiments across multiple image modalities showcase notable enhancements in performance compared to existing techniques.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 66 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

42 of 66 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training20 Oct
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
  3. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study30 Sep

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
27 more
  1. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  2. 2021Grounded Language-Image Pre-trainingcited by 1,147
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  4. 2019Publicly Available Clinical BERT Embeddingscited by 995
  5. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  7. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  8. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  9. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  10. 2018A Survey on Deep Transfer Learningcited by 417
  11. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  12. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  13. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  14. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  15. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  16. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  17. 2019Rank consistent ordinal regression for neural networks with application to age estimationcited by 132
  18. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  19. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  20. 2020COVID-19 Image Data Collection: Prospective Predictions Are the Futurecited by 100
  21. 2020Annotation-efficient deep learning for automatic medical image segmentationcited by 79
  22. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  23. 2021Every Annotation Counts: Multi-label Deep Supervision for Medical Image Segmentationcited by 37
  24. 2023High Fidelity 3D Hand Shape Reconstruction via Scalable Graph Frequency Decompositioncited by 18
  25. 2022When Neural Networks Fail to Generalize? A Model Sensitivity Perspectivecited by 10
  26. 2024Spectrum AUC Difference (SAUCD): Human-aligned 3D Shape Evaluationcited by 9
  27. 2023Spectral Adversarial MixUp for Few-Shot Unsupervised Domain Adaptationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image Analysiscited by 3
  2. 2025Anatomy-Aware Low-Dose CT Denoising via Pretrained Vision Models and Semantic-Guided Contrastive Learningcited by 2
  3. 2025CLIP-IT: CLIP-based Pairing for Histology Images Classificationcited by 1
  4. 2025Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridgingcited by 0
  5. 2025CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistencycited by 0
  6. 2026DECIS: Dual-Evidence Corrective Verification for Interpretable Strabismus Diagnostic Decision-Makingcited by 0
  7. 2026RaLMPH: Reliability-aware Learning for Multi-Pathologist Harmonization in Whole-Slide Image Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  2. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classification
  3. 2024Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training
  4. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
  5. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework
  6. 2024Few-shot Adaptation of Medical Vision-Language Models
  7. 2024Aligning Medical Images with General Knowledge from Large Language Models
  8. 2025Few-Shot, Now for Real: Medical VLMs Adaptation without Balanced Sets or Validation