mlsanityData to 7 Oct 2026

Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging

arXiv 2401.02565 ↗First posted 4 Jan 2024eess.IV cs.CV q-bio.TOcited by 9

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

In the context of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. Leveraging the Kather Colon dataset with 7,180 H&E images across nine tissue types, our investigation employs Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The outcomes reveal a 100% success rate in manipulating PLIP's predictions, underscoring its susceptibility to adversarial perturbations. The qualitative analysis of adversarial examples delves into the interpretability challenges, shedding light on nuanced changes in predictions induced by adversarial manipulations. These findings contribute crucial insights into the interpretability, domain adaptation, and trustworthiness of Vision Language Models in medical imaging. The study emphasizes the pressing need for robust defenses to ensure the reliability of AI models. The source codes for this experiment can be found at https://github.com/jaiprakash1824/VLM_Adv_Attack.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 16 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

10 of 16 references.

References · 10

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  5. 2022Diffusion Models for Adversarial Purificationcited by 469
  6. 2020To be Robust or to be Fair: Towards Fairness in Adversarial Trainingcited by 137
  7. 2020Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primarycited by 89
  8. 2020Deep learning-based transformation of the H&E stain into special stainscited by 55
  9. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  10. 2021Online Adversarial Purification based on Self-Supervisioncited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  2. 2026SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Carecited by 5
  3. 2024Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Modelscited by 4
  4. 2025Universal and Transferable Attacks on Pathology Foundation Modelscited by 2
  5. 2025Hammering the Diagnosis: Rowhammer-Induced Stealthy Trojan Attacks on ViT-Based Medical Imagingcited by 1
  6. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  7. 2025The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Modelscited by 0
  8. 2025Cross-Domain Image Synthesis: Generating H&E from Multiplex Biomarker Imagingcited by 0
  9. 2026Stain-Aware Wavelet Regularization for Instant Adversarial Purification in Histopathologycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
  2. 2025VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models
  3. 2024Prompt Injection Attacks on Large Language Models in Oncology
  4. 2026When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
  5. 2026When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound
  6. 20256 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models
  7. 2025Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment
  8. 2025SafeMed-R1: Adversarial Reinforcement Learning for Generalizable and Robust Medical Reasoning in Vision-Language Models