VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Vision Language Models (VLMs) hold great promise for streamlining labour-intensive medical imaging workflows, yet systematic security evaluations in clinical settings remain scarce. We introduce VSF--Med, an end-to-end vulnerability-scoring framework for medical VLMs that unites three novel components: (i) a rich library of sophisticated text-prompt attack templates targeting emerging threat vectors; (ii) imperceptible visual perturbations calibrated by structural similarity (SSIM) thresholds to preserve clinical realism; and (iii) an eight-dimensional rubric evaluated by two independent judge LLMs, whose raw scores are consolidated via z-score normalization to yield a 0--32 composite risk metric. Built entirely on publicly available datasets and accompanied by open-source code, VSF--Med synthesizes over 30,000 adversarial variants from 5,000 radiology images and enables reproducible benchmarking of any medical VLM with a single command. Our consolidated analysis reports mean z-score shifts of 0.90σ for persistence-of-attack-effects, 0.74σ for prompt-injection effectiveness, and 0.63σ for safety-bypass success across state-of-the-art VLMs. Notably, Llama-3.2-11B-Vision-Instruct exhibits a peak vulnerability increase of 1.29σ for persistence-of-attack-effects, while GPT-4o shows increases of 0.69σ for that same vector and 0.28σ for prompt-injection attacks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 51 references that are in the map. First posted on arXiv 25 Jun 2025.
Builds on
31 of 51 references.
Closest ancestors · 6
Same direction, newest first
- 2024FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks1 Oct
- 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug
- 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models10 Jun
- 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models14 Feb
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
- 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May
Other references · 25
- 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
- 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
- 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
- 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
- 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
- 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
- 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 65
- 2024FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Modelscited by 47
- 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
- 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
- 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
- 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 28
13 more
- 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectorscited by 26
- 2018Unsupervised Multimodal Representation Learning across Medical Images and Reportscited by 15
- 2024Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architecturescited by 12
- 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 8
- 2024Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vectorcited by 8
- 2024Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Modelscited by 4
- 2024DIRI: Adversarial Patient Reidentification with Large Language Models for Evaluating Clinical Text Anonymizationcited by 4
- 2024Fighting Fire with Fire: Adversarial Prompting to Generate a Misinformation Detection Datasetcited by 3
- 2025CoRPA: Adversarial Image Generation for Chest X-rays Using Concept Vector Perturbations and Generative Modelscited by 3
- 2025CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformationcited by 3
- 2024Adversarial Databases Improve Success in Retrieval-based Large Language Modelscited by 1
- 2025A Framework for Evaluating Vision-Language Model Safety: Building Trust in AI for Public Sector Applicationscited by 1
- 2025Security-First AI: Foundations for Robust and Trustworthy Systemscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Prompt Injection Attacks on Large Language Models in Oncology
- 2026When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
- 2025Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
- 2025Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations
- 2026CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models
- 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
- 2024Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging
- 2025MedFoundationHub: A Lightweight and Secure Toolkit for Deploying Medical Vision Language Foundation Models