mlsanityData to 7 Oct 2026

Aligning Medical Images with General Knowledge from Large Language Models

arXiv 2409.00341 ↗First posted 31 Aug 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Pre-trained large vision-language models (VLMs) like CLIP have revolutionized visual representation learning using natural language as supervisions, and demonstrated promising generalization ability. In this work, we propose ViP, a novel visual symptom-guided prompt learning framework for medical image analysis, which facilitates general knowledge transfer from CLIP. ViP consists of two key components: a visual symptom generator (VSG) and a dual-prompt network. Specifically, VSG aims to extract explicable visual symptoms from pre-trained large language models, while the dual-prompt network utilizes these visual symptoms to guide the training on two learnable prompt modules, i.e., context prompt and merge prompt, which effectively adapts our framework to medical image analysis via large VLMs. Extensive experimental results demonstrate that ViP can outperform state-of-the-art methods on two challenging datasets.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 27 references that are in the map. First posted on arXiv 31 Aug 2024.

Builds on

21 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Exploring Low-Resource Medical Image Classification with Weakly Supervised Prompt Learning6 Feb
  2. 2023Few-shot medical image classification with simple shape and texture text descriptors using vision-language models8 Aug
  3. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study30 Sep

Other references · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  10. 2022Visual Classification via Description from Large Language Modelscited by 331
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
6 more
  1. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  2. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  3. 2023A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosiscited by 30
  4. 2023Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosiscited by 26
  5. 2023Rethinking Boundary Detection in Deep Learning Models for Medical Image Segmentationcited by 9
  6. 2023Boosting Convolution with Efficient MLP-Permutation for Volumetric Medical Image Segmentationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Modelscited by 28
  2. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classificationcited by 5
  3. 2025PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentationcited by 2
  4. 2025NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understandingcited by 0
  5. 2026Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classificationcited by 0
  6. 2026Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentationcited by 0
  7. 2026Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations
  2. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  3. 2024Disease-informed Adaptation of Vision-Language Models
  4. 2026Representation-guided in-context learning for medical image interpretation with multimodal large language models
  5. 2025Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models
  6. 2024XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization
  7. 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
  8. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study