How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Recent advancements in medical vision-language pre-training (MedVLP) have significantly enhanced zero-shot medical vision tasks such as image classification by leveraging large-scale medical image-text pair pre-training. However, the performance of these tasks can be heavily influenced by the variability in textual prompts describing the categories, necessitating robustness in MedVLP models to diverse prompt styles. Yet, this sensitivity remains underexplored. In this work, we are the first to systematically assess the sensitivity of three widely-used MedVLP methods to a variety of prompts across 15 different diseases. To achieve this, we designed six unique prompt styles to mirror real clinical scenarios, which were subsequently ranked by interpretability. Our findings indicate that all MedVLP models evaluated show unstable performance across different prompt styles, suggesting a lack of robustness. Additionally, the models' performance varied with increasing prompt interpretability, revealing difficulties in comprehending complex medical concepts. This study underscores the need for further development in MedVLP methodologies to enhance their robustness to diverse zero-shot prompts.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 32 references that are in the map. First posted on arXiv 31 Aug 2024.
Builds on
29 of 32 references.
Closest ancestors · 9
Same direction, newest first
- 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework12 Mar
- 2024Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-training2 Jan
- 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency3 Dec
- 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training3 Dec
- 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training11 Oct
- 2023Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images10 Oct
- 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization17 Jul
- 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation7 Jun
- 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias31 May
Other references · 20
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
- 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
- 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
- 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
- 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
- 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
- 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
- 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
- 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
8 more
- 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
- 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
- 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
- 2022Improving Zero-shot Generalization and Robustness of Multi-modal Modelscited by 44
- 2024Learning to Prompt with Text Only Supervision for Vision-Language Modelscited by 37
- 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
- 2023Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosiscited by 23
- 2024Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Followingcited by 13
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
- 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
- 2024Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?
- 2023Medical Vision Language Pretraining: A survey
- 2025Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations
- 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning
- 2025Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
- 2026How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images