mlsanityData to 14 Oct 2026

Universal Adversarial Perturbations for Vision-Language Pre-trained Models

arXiv 2405.05524 ↗First posted 9 May 2024cs.CV cs.MMcited by 31

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language pre-trained (VLP) models have been the foundation of numerous vision-language tasks. Given their prevalence, it becomes imperative to assess their adversarial robustness, especially when deploying them in security-crucial real-world applications. Traditionally, adversarial perturbations generated for this assessment target specific VLP models, datasets, and/or downstream tasks. This practice suffers from low transferability and additional computation costs when transitioning to new scenarios. In this work, we thoroughly investigate whether VLP models are commonly sensitive to imperceptible perturbations of a specific pattern for the image modality. To this end, we propose a novel black-box method to generate Universal Adversarial Perturbations (UAPs), which is so called the Effective and T ransferable Universal Adversarial Attack (ETU), aiming to mislead a variety of existing VLP models in a range of downstream tasks. The ETU comprehensively takes into account the characteristics of UAPs and the intrinsic cross-modal interactions to generate effective UAPs. Under this regime, the ETU encourages both global and local utilities of UAPs. This benefits the overall utility while reducing interactions between UAP units, improving the transferability. To further enhance the effectiveness and transferability of UAPs, we also design a novel data augmentation method named ScMix. ScMix consists of self-mix and cross-mix data transformations, which can effectively increase the multi-modal data diversity while preserving the semantics of the original data. Through comprehensive experiments on various downstream tasks, VLP models, and datasets, we demonstrate that the proposed method is able to achieve effective and transferrable universal adversarial attacks.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 39 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

25 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 23

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2016Adversarial examples in the physical worldcited by 1,769
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2016Universal adversarial perturbationscited by 784
  12. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
11 more
  1. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  2. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  3. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  4. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
  5. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  6. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
  7. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  8. 2021Admix: Enhancing the Transferability of Adversarial Attackscited by 156
  9. 2017Generative Adversarial Perturbationscited by 153
  10. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  11. 2023Zero-Shot Learning by Harnessing Adversarial Samplescited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 13.

31 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  2. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 40
  3. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 32
  4. 2025Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Frameworkcited by 24
  5. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 23
  6. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  7. 2025One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIPcited by 15
  8. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  9. 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questionscited by 14
  10. 2024Adversarial Backdoor Defense in CLIPcited by 10
  11. 2024Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Modelscited by 4
  12. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 4
19 more
  1. 2025EmoRAG: Evaluating RAG Robustness to Symbolic Perturbationscited by 3
  2. 2025MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Modelscited by 2
  3. 2025Attacking Attention of Foundation Models Disrupts Downstream Taskscited by 2
  4. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  5. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  6. 2025PATFinger: Prompt-Adapted Transferable Fingerprinting against Unauthorized Multimodal Dataset Usagecited by 1
  7. 2026Universal Adversarial Attacks against Closed-Source MLLMs via Target-View Routed Meta Optimizationcited by 1
  8. 2024Attacks on multimodal modelscited by 0
  9. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Modelscited by 0
  10. 2026Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Modelscited by 0
  11. 2026Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluationcited by 0
  12. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  13. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  14. 2026HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Modelscited by 0
  15. 2026From Clouds to Hallucinations: Atmospheric Retrieval Hijacking in Remote Sensing Vision-Language RAGcited by 0
  16. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  17. 2026DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Modelscited by 0
  18. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipelinecited by 0
  19. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models
  2. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  3. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  4. 2026VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models
  5. 2026Breaking the weakest link to evade vision language models
  6. 2026Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics
  7. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation
  8. 2026A Two-Stage Globally-Diverse Adversarial Attack for Vision-Language Pre-training Models