mlsanityData to 14 Oct 2026

Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models

arXiv 2408.02980 ↗First posted 6 Aug 2024cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Recent studies on AI security have highlighted the vulnerability of Vision-Language Pre-training (VLP) models to subtle yet intentionally designed perturbations in images and texts. Investigating multimodal systems' robustness via adversarial attacks is crucial in this field. Most multimodal attacks are sample-specific, generating a unique perturbation for each sample to construct adversarial samples. To the best of our knowledge, it is the first work through multimodal decision boundaries to explore the creation of a universal, sample-agnostic perturbation that applies to any image. Initially, we explore strategies to move sample points beyond the decision boundaries of linear classifiers, refining the algorithm to ensure successful attacks under the top k accuracy metric. Based on this foundation, in visual-language tasks, we treat visual and textual modalities as reciprocal sample points and decision hyperplanes, guiding image embeddings to traverse text-constructed decision boundaries, and vice versa. This iterative process consistently refines a universal perturbation, ultimately identifying a singular direction within the input space which is exploitable to impair the retrieval performance of VLP models. The proposed algorithms support the creation of global perturbations or adversarial patches. Comprehensive experiments validate the effectiveness of our method, showcasing its data, task, and model transferability across various VLP models and datasets. Code: https://github.com/LibertazZ/MUAP

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 44 references that are in the map. First posted on arXiv 6 Aug 2024.

Builds on

23 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models25 Jul
  2. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models21 Feb
  3. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2016Universal adversarial perturbationscited by 784
  12. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
8 more
  1. 2017Generative Adversarial Perturbationscited by 153
  2. 2018LaVAN: Localized and Visible Adversarial Noisecited by 113
  3. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  4. 2017Fast Feature Fool: A data independent approach to universal adversarial perturbationscited by 66
  5. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
  6. 2018Ask, Acquire, and Attack: Data-free UAP Generation using Class Impressionscited by 31
  7. 2024Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learningcited by 10
  8. 2020Upgrading the Newsroom: An Automated Image Selection System for News Articlescited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  2. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Surveycited by 23

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  2. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models
  3. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models
  4. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  5. 2026Breaking the weakest link to evade vision language models
  6. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack
  7. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models
  8. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach