# VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models

arXiv: https://arxiv.org/abs/2507.00052 · HTML: https://mlsanity.com/paper/2507.00052/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 31)

- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems](https://mlsanity.com/paper/2410.07283/) (2024)
- [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025)
- [Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt](https://mlsanity.com/paper/2406.04031/) (2024)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI](https://mlsanity.com/paper/2408.03361/) (2024)
- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations](https://mlsanity.com/paper/2502.14881/) (2025)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models](https://mlsanity.com/paper/2406.06007/) (2024)
- [Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications](https://mlsanity.com/paper/2401.07612/) (2024)
- [FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models](https://mlsanity.com/paper/2407.00983/) (2024)
- [Denial-of-Service Poisoning Attacks against Large Language Models](https://mlsanity.com/paper/2410.10760/) (2024)
- [Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settings](https://mlsanity.com/paper/2412.13879/) (2024)
- [An Early Categorization of Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2402.00898/) (2024)
- [MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models](https://mlsanity.com/paper/2402.09262/) (2024)
- [Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors](https://mlsanity.com/paper/2405.10529/) (2024)
- [Unsupervised Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/1811.08615/) (2018)
- [Systematically Analyzing Prompt Injection Vulnerabilities in Diverse LLM Architectures](https://mlsanity.com/paper/2410.23308/) (2024)
- [FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks](https://mlsanity.com/paper/2410.01089/) (2024)
- [Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models](https://mlsanity.com/paper/2405.20775/) (2024)
- [Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector](https://mlsanity.com/paper/2410.22888/) (2024)
- [Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models](https://mlsanity.com/paper/2407.02716/) (2024)
- [DIRI: Adversarial Patient Reidentification with Large Language Models for Evaluating Clinical Text Anonymization](https://mlsanity.com/paper/2410.17035/) (2024)
- [Fighting Fire with Fire: Adversarial Prompting to Generate a Misinformation Detection Dataset](https://mlsanity.com/paper/2401.04481/) (2024)
- [CoRPA: Adversarial Image Generation for Chest X-rays Using Concept Vector Perturbations and Generative Models](https://mlsanity.com/paper/2502.05214/) (2025)
- [CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation](https://mlsanity.com/paper/2505.01900/) (2025)
- [Adversarial Databases Improve Success in Retrieval-based Large Language Models](https://mlsanity.com/paper/2407.14609/) (2024)
- [A Framework for Evaluating Vision-Language Model Safety: Building Trust in AI for Public Sector Applications](https://mlsanity.com/paper/2502.16361/) (2025)

## Built on by (most cited first, 1 of 1)

- [6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models](https://mlsanity.com/paper/2512.04238/) (2025)

## Abstract (arXiv)

Vision Language Models (VLMs) hold great promise for streamlining labour-intensive medical imaging workflows, yet systematic security evaluations in clinical settings remain scarce. We introduce VSF--Med, an end-to-end vulnerability-scoring framework for medical VLMs that unites three novel components: (i) a rich library of sophisticated text-prompt attack templates targeting emerging threat vectors; (ii) imperceptible visual perturbations calibrated by structural similarity (SSIM) thresholds to preserve clinical realism; and (iii) an eight-dimensional rubric evaluated by two independent judge LLMs, whose raw scores are consolidated via z-score normalization to yield a 0--32 composite risk metric. Built entirely on publicly available datasets and accompanied by open-source code, VSF--Med synthesizes over 30,000 adversarial variants from 5,000 radiology images and enables reproducible benchmarking of any medical VLM with a single command. Our consolidated analysis reports mean z-score shifts of 0.90σ for persistence-of-attack-effects, 0.74σ for prompt-injection effectiveness, and 0.63σ for safety-bypass success across state-of-the-art VLMs. Notably, Llama-3.2-11B-Vision-Instruct exhibits a peak vulnerability increase of 1.29σ for persistence-of-attack-effects, while GPT-4o shows increases of 0.69σ for that same vector and 0.28σ for prompt-injection attacks.