mlsanityData to 7 Oct 2026

Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory

arXiv 2403.12445 ↗First posted 19 Mar 2024cs.CVcited by 35

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language pre-training (VLP) models exhibit remarkable capabilities in comprehending both images and text, yet they remain susceptible to multimodal adversarial examples (AEs). Strengthening attacks and uncovering vulnerabilities, especially common issues in VLP models (e.g., high transferable AEs), can advance reliable and practical VLP models. A recent work (i.e., Set-level guidance attack) indicates that augmenting image-text pairs to increase AE diversity along the optimization path enhances the transferability of adversarial examples significantly. However, this approach predominantly emphasizes diversity around the online adversarial examples (i.e., AEs in the optimization period), leading to the risk of overfitting the victim model and affecting the transferability. In this study, we posit that the diversity of adversarial examples towards the clean input and online AEs are both pivotal for enhancing transferability across VLP models. Consequently, we propose using diversification along the intersection region of adversarial trajectory to expand the diversity of AEs. To fully leverage the interaction between modalities, we introduce text-guided adversarial example selection during optimization. Furthermore, to further mitigate the potential overfitting, we direct the adversarial text deviating from the last intersection region along the optimization path, rather than adversarial images as in existing methods. Extensive experiments affirm the effectiveness of our method in improving transferability across various VLP models and downstream vision-and-language tasks.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 66 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

54 of 66 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models14 Mar
  2. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  3. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  4. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  5. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
37 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  7. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  8. 2021Intriguing Properties of Vision Transformerscited by 446
  9. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  10. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  11. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  13. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  14. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  15. 2021Admix: Enhancing the Transferability of Adversarial Attackscited by 155
  16. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  17. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  18. 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
  19. 2020Efficient Adversarial Attacks for Visual Object Trackingcited by 66
  20. 2020Adv-watermark: A Novel Watermark Perturbation for Adversarial Examplescited by 56
  21. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
  22. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 43
  23. 2023Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Modelscited by 42
  24. 2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
  25. 2023CDUL: CLIP-Driven Unsupervised Learning for Multi-Label Image Classificationcited by 39
  26. 2023Improving the Transferability of Adversarial Samples by Path-Augmented Methodcited by 36
  27. 2020Improving Query Efficiency of Black-box Adversarial Attackcited by 34
  28. 2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 34
  29. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  30. 2021PAC-Bayes Information Bottleneckcited by 28
  31. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  32. 2021Learning to Adversarially Blur Visual Object Trackingcited by 14
  33. 2022ALA: Naturalness-aware Adversarial Lightness Attackcited by 11
  34. 2024Hard-label based Small Query Black-box Adversarial Attackcited by 10
  35. 2021AdvFilter: Predictive Perturbation-aware Filtering against Adversarial Attack via Multi-domain Learningcited by 9
  36. 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9
  37. 2021Dense Contrastive Visual-Linguistic Pretrainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 13; 2026: 13.

35 papers.

Sort:

By area:

  1. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  2. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  3. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  4. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  5. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  6. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  7. 2024Towards Transferable Attacks Against Vision-LLMs in Autonomous Driving with Typographycited by 17
  8. 2024Texture Re-scalable Universal Adversarial Perturbationcited by 9
  9. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  10. 2026Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Securitycited by 8
  11. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  12. 2025Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Modelscited by 6
23 more
  1. 2024Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Modelscited by 4
  2. 2025Adversarial-Guided Diffusion for Multimodal LLM Attackscited by 3
  3. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 3
  4. 2025MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Modelscited by 2
  5. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  6. 2024MAGIC: Mastering Physical Adversarial Generation in Context through Collaborative LLM Agentscited by 1
  7. 2025Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updatescited by 1
  8. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1
  9. 2025Adversarial Video Promotion Against Text-to-Video Retrievalcited by 1
  10. 2026Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMscited by 1
  11. 2026Frequency-Domain Regularized Adversarial Alignment for Transferable Attacks against Closed-Source MLLMscited by 1
  12. 2024Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Modelscited by 0
  13. 2025AngleRoCL: Angle-Robust Concept Learning for Physically View-Invariant T2I Adversarial Patchescited by 0
  14. 2025Vulnerability-Aware Robust Multimodal Adversarial Trainingcited by 0
  15. 2026A Two-Stage Globally-Diverse Adversarial Attack for Vision-Language Pre-training Modelscited by 0
  16. 2026The Power of Decaying Steps: Enhancing Attack Stability and Transferability for Sign-based Optimizerscited by 0
  17. 2026Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluationcited by 0
  18. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interactioncited by 0
  19. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  20. 2026HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Modelscited by 0
  21. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correctioncited by 0
  22. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipelinecited by 0
  23. 2026Multimodal Plant Root Phenotyping with Integration of 3D Skeleton Extraction and Language Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  2. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack
  3. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  4. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  5. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
  6. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models
  7. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  8. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models