mlsanityData to 7 Oct 2026

Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models

arXiv 2405.20090 ↗First posted 30 May 2024cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) demonstrate exceptional performance in cross-modality interaction, yet they also suffer adversarial vulnerabilities. In particular, the transferability of adversarial examples remains an ongoing challenge. In this paper, we specifically analyze the manifestation of adversarial transferability among MLLMs and identify the key factors that influence this characteristic. We discover that the transferability of MLLMs exists in cross-LLM scenarios with the same vision encoder and indicate underlinetextittwo key Factors that may influence transferability. We provide two semantic-level data augmentation methods, Adding Image Patch (AIP) and Typography Augment Transferability Method (TATM), which boost the transferability of adversarial examples across MLLMs. To explore the potential impact in the real world, we utilize two tasks that can have both negative and positive societal impacts: ding182 Harmful Content Insertion and ding183 Information Protection.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 82 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

53 of 82 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models14 Mar
  2. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  3. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  4. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  5. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  6. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  7. 2023Defense-Prefix for Preventing Typographic Attacks on CLIP10 Apr

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
34 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  6. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  7. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  9. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  10. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  11. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  12. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  13. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  14. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 242
  15. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  16. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  17. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  18. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  19. 2021Admix: Enhancing the Transferability of Adversarial Attackscited by 155
  20. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 67
  21. 2023Structure Invariant Transformation for better Adversarial Transferabilitycited by 61
  22. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  23. 2023Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?cited by 37
  24. 2023Improving the Transferability of Adversarial Samples by Path-Augmented Methodcited by 36
  25. 2022Transferable Adversarial Attack based on Integrated Gradientscited by 35
  26. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  27. 2021Survey: Image Mixing and Deleting for Data Augmentationcited by 25
  28. 2023Transferring Foundation Models for Generalizable Robotic Manipulationcited by 18
  29. 2023Improving the Transferability of Adversarial Examples with Arbitrary Style Transfercited by 13
  30. 2024The New Agronomists: Language Models are Experts in Crop Managementcited by 12
  31. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  32. 2023Improving Adversarial Transferability via Model Alignmentcited by 10
  33. 2024VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Modelscited by 10
  34. 2021Controlled Caption Generation for Images Through Adversarial Attackscited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2025Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Modelscited by 17
  2. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  3. 2024Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agentscited by 5
  4. 2025Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attackcited by 1
  5. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approach
  2. 2025Survey of Adversarial Robustness in Multimodal Large Language Models
  3. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey
  4. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization
  5. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory
  6. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  7. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack
  8. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models