mlsanityData to 14 Oct 2026

Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation

arXiv 2505.02971 ↗First posted 5 May 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. footnotehttps://github.com/anjilab/secure-private-ai

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 33 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

20 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks10 May
  2. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models15 Aug

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 589
  7. 2021Image Segmentation Using Text and Image Promptscited by 512
  8. 2017Deep Learning for Medical Image Analysiscited by 401
  9. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  10. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  11. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  12. 2023GRES: Generalized Referring Expression Segmentationcited by 254
6 more
  1. 2019Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiographycited by 244
  2. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 198
  3. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  4. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 153
  5. 2018Adversarial Attacks Against Medical Deep Learning Systemscited by 62
  6. 2023Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiographycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Modelscited by 2
  2. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging
  2. 2024On Evaluating Adversarial Robustness of Volumetric Medical Segmentation Models
  3. 2025VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models
  4. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning
  5. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  6. 2024Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models
  7. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  8. 2025Robust Fairness Vision-Language Learning for Medical Image Analysis