# Aligning Medical Images with General Knowledge from Large Language Models

arXiv: https://arxiv.org/abs/2409.00341 · HTML: https://mlsanity.com/paper/2409.00341/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 31 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study](https://mlsanity.com/paper/2209.15517/) (2022)
- [AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation](https://mlsanity.com/paper/2203.10095/) (2022)
- [Bayesian Prompt Learning for Image-Language Model Generalization](https://mlsanity.com/paper/2210.02390/) (2022)
- [A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis](https://mlsanity.com/paper/2307.01981/) (2023)
- [Coherent Concept-based Explanations in Medical Image and Its Application to Skin Lesion Diagnosis](https://mlsanity.com/paper/2304.04579/) (2023)
- [Exploring Low-Resource Medical Image Classification with Weakly Supervised Prompt Learning](https://mlsanity.com/paper/2402.03783/) (2024)
- [Rethinking Boundary Detection in Deep Learning Models for Medical Image Segmentation](https://mlsanity.com/paper/2305.00678/) (2023)
- [Few-shot medical image classification with simple shape and texture text descriptors using vision-language models](https://mlsanity.com/paper/2308.04005/) (2023)
- [Boosting Convolution with Efficient MLP-Permutation for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2303.13111/) (2023)

## Built on by (most cited first, 8 of 8)

- [BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models](https://mlsanity.com/paper/2411.15232/) (2024)
- [Medical Knowledge Intervention Prompt Tuning for Medical Image Classification](https://mlsanity.com/paper/2511.12639/) (2025)
- [PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation](https://mlsanity.com/paper/2503.18227/) (2025)
- [NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding](https://mlsanity.com/paper/2508.04101/) (2025)
- [Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification](https://mlsanity.com/paper/2604.23977/) (2026)
- [Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2604.26221/) (2026)
- [Boosting Ultrasound Image Classification via Attribute-Guided Dual-Branch Framework](https://mlsanity.com/paper/2607.01648/) (2026)
- [Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classification](https://mlsanity.com/paper/2609.13303/) (2026)

## Abstract (arXiv)

Pre-trained large vision-language models (VLMs) like CLIP have revolutionized visual representation learning using natural language as supervisions, and demonstrated promising generalization ability. In this work, we propose ViP, a novel visual symptom-guided prompt learning framework for medical image analysis, which facilitates general knowledge transfer from CLIP. ViP consists of two key components: a visual symptom generator (VSG) and a dual-prompt network. Specifically, VSG aims to extract explicable visual symptoms from pre-trained large language models, while the dual-prompt network utilizes these visual symptoms to guide the training on two learnable prompt modules, i.e., context prompt and merge prompt, which effectively adapts our framework to medical image analysis via large VLMs. Extensive experimental results demonstrate that ViP can outperform state-of-the-art methods on two challenging datasets.