mlsanityData to 7 Oct 2026

One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models

arXiv 2406.05491 ↗First posted 8 Jun 2024cs.CV cs.CRcited by 39

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Pre-training (VLP) models have exhibited unprecedented capability in many applications by taking full advantage of the multimodal alignment. However, previous studies have shown they are vulnerable to maliciously crafted adversarial samples. Despite recent success, these methods are generally instance-specific and require generating perturbations for each input sample. In this paper, we reveal that VLP models are also vulnerable to the instance-agnostic universal adversarial perturbation (UAP). Specifically, we design a novel Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to achieve the attack. In light that the pivotal multimodal alignment is achieved through the advanced contrastive learning technique, we devise to turn this powerful weapon against themselves, i.e., employ a malicious version of contrastive learning to train the C-PGC based on our carefully crafted positive and negative image-text pairs for essentially destroying the alignment relationship learned by VLP models. Besides, C-PGC fully utilizes the characteristics of Vision-and-Language (V+L) scenarios by incorporating both unimodal and cross-modal information as effective guidance. Extensive experiments show that C-PGC successfully forces adversarial samples to move away from their original area in the VLP model's feature space, thus essentially enhancing attacks across various victim models and V+L tasks. The GitHub repository is available at https://github.com/ffhibnese/CPGC_VLP_Universal_Attacks.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 102 of its 136 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

102 of 136 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Models16 May
  2. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  3. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  4. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 98

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2016Modeling Context in Referring Expressionscited by 1,054
  13. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  14. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  15. 2016Adversarial Machine Learning at Scalecited by 934
  16. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  17. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  18. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  19. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  20. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  21. 2016Universal adversarial perturbationscited by 774
  22. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  23. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  24. 2017Countering Adversarial Images using Input Transformationscited by 517
  25. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  28. 2022Diffusion Models for Adversarial Purificationcited by 469
  29. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  30. 2017Mitigating Adversarial Effects Through Randomizationcited by 382
  31. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  32. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  33. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  34. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  35. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  36. 2019Simple Black-box Adversarial Attackscited by 249
  37. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  38. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 17; 2026: 14.

39 papers.

Sort:

By area:

  1. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  2. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  3. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  4. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  5. 2024CLIP-Guided Generative Networks for Transferable Targeted Adversarial Attackscited by 19
  6. 2024A Closer Look at GAN Priors: Exploiting Intermediate Features for Enhanced Model Inversion Attackscited by 17
  7. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMscited by 16
  8. 2024Hierarchical Features Matter: A Deep Exploration of Progressive Parameterization Method for Dataset Distillationcited by 13
  9. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  10. 2024Video Watermarking: Safeguarding Your Video from (Unauthorized) Annotations by Video-based LLMscited by 7
  11. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  12. 2025Protecting Your Video Content: Disrupting Automated Video-based LLM Annotationscited by 5
27 more
  1. 2024Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Modelscited by 4
  2. 2024Editable-DeepSC: Reliable Cross-Modal Semantic Communications for Facial Editingcited by 3
  3. 2025CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMscited by 3
  4. 2026Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderationcited by 3
  5. 2025Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytellingcited by 2
  6. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  7. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  8. 2026Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimizationcited by 2
  9. 2026Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generationcited by 2
  10. 2026Bypassing Copyright Protection in Diffusion-based Customization via Two-Stage Latent Feature Optimizationcited by 2
  11. 2025Cross-Model Transferability of Adversarial Patches in Real-time Segmentation for Autonomous Drivingcited by 1
  12. 2025Learning Fourier shapes to probe the geometric world of deep neural networkscited by 1
  13. 2026Universal Anti-forensics Attack against Image Forgery Detection via Multi-modal Guidancecited by 1
  14. 2025Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIPcited by 0
  15. 2025Towards Cross-Domain Multi-Targeted Adversarial Attackscited by 0
  16. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Modelscited by 0
  17. 2025Revisiting the Privacy Risks of Split Inference: A GAN-Based Data Reconstruction Attack via Progressive Feature Optimizationcited by 0
  18. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Modelscited by 0
  19. 2025Semantic Router: On the Feasibility of Hijacking MLLMs via a Single Adversarial Perturbationcited by 0
  20. 2026Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Modelscited by 0
  21. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  22. 2026Adversarial Attacks Against MLLMs via Progressive Resolution Processing and Adaptive Feature Alignmentcited by 0
  23. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  24. 2026DarkLLM: Learning Language-Driven Adversarial Attacks with Large Language Modelscited by 0
  25. 2026CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignmentcited by 0
  26. 2026One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMscited by 0
  27. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  2. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
  3. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
  4. 2026Breaking the weakest link to evade vision language models
  5. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  6. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  7. 2024Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models
  8. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models