mlsanityData to 14 Oct 2026

Adversarial attacks against Modern Vision-Language Models

arXiv 2603.16960 ↗First posted 17 Mar 2026cs.CR cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 18 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

10 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector19 Aug

Other references · 9

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2016Adversarial examples in the physical worldcited by 1,769
  4. 2016Adversarial Machine Learning at Scalecited by 936
  5. 2017Adversarial Patchcited by 517
  6. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  7. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 89
  8. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
  9. 2024VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generationcited by 0
  2. 2026Transferable Spatial Temporal Coherence Adversarial Attack on Black-Box Vision Language Models for Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  2. 2025A Framework for Evaluating Vision-Language Model Safety: Building Trust in AI for Public Sector Applications
  3. 2026Breaking the weakest link to evade vision language models
  4. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  5. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models
  6. 2024Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation
  7. 2025Adversarial Attacks on Robotic Vision Language Action Models
  8. 2025AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models