mlsanityData to 7 Oct 2026

An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models

arXiv 2403.09766 ↗First posted 14 Mar 2024cs.CVcited by 40

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Different from traditional task-specific vision models, recent large VLMs can readily adapt to different vision tasks by simply using different textual instructions, i.e., prompts. However, a well-known concern about traditional task-specific vision models is that they can be misled by imperceptible adversarial perturbations. Furthermore, the concern is exacerbated by the phenomenon that the same adversarial perturbations can fool different task-specific models. Given that VLMs rely on prompts to adapt to different tasks, an intriguing question emerges: Can a single adversarial image mislead all predictions of VLMs when a thousand different prompts are given? This question essentially introduces a novel perspective on adversarial transferability: cross-prompt adversarial transferability. In this work, we propose the Cross-Prompt Attack (CroPA). This proposed method updates the visual adversarial perturbation with learnable prompts, which are designed to counteract the misleading effects of the adversarial image. By doing this, CroPA significantly improves the transferability of adversarial examples across prompts. Extensive experiments are conducted to verify the strong cross-prompt adversarial transferability of CroPA with prevalent VLMs including Flamingo, BLIP-2, and InstructBLIP in various different tasks. Our source code is available at urlhttps://github.com/Haochen-Luo/CroPA.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 46 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

38 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  2. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
24 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2016Universal adversarial perturbationscited by 774
  3. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  4. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  5. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  6. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  7. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  8. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  9. 2018Exploring Visual Relationship for Image Captioningcited by 215
  10. 2017The Space of Transferable Adversarial Examplescited by 156
  11. 2019Cross-Domain Transferability of Adversarial Perturbationscited by 92
  12. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  13. 2022SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustnesscited by 74
  14. 2017Fast Feature Fool: A data independent approach to universal adversarial perturbationscited by 65
  15. 2021Human-Adversarial Visual Question Answeringcited by 57
  16. 2019Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reductioncited by 47
  17. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 43
  18. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 30
  19. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  20. 2019Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variablescited by 26
  21. 2018Task-generalizable Adversarial Attack based on Perceptual Metriccited by 18
  22. 2023Reliable Evaluation of Adversarial Transferabilitycited by 12
  23. 2021Controlled Caption Generation for Images Through Adversarial Attackscited by 9
  24. 2022longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attackscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 12; 2026: 10.

40 papers.

Sort:

By area:

  1. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  2. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  3. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  4. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  5. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  6. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 35
  7. 2024Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Modelscited by 29
  8. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  9. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  10. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  11. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 16
  12. 2024Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samplescited by 14
28 more
  1. 2024Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Modelscited by 11
  2. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  3. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimizationcited by 11
  4. 2025MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attackscited by 10
  5. 2024Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Modelscited by 8
  6. 2024Scaling Laws for Black box Adversarial Attackscited by 8
  7. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Modelscited by 8
  8. 2025Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approachcited by 7
  9. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  10. 2024Refusing Safe Prompts for Multi-modal Large Language Modelscited by 6
  11. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Modelscited by 5
  12. 2024Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attackscited by 5
  13. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  14. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  15. 2025Towards Transferable Defense Against Malicious Image Editscited by 2
  16. 2026Visual Memory Injection Attacks for Multi-Turn Conversationscited by 2
  17. 2025Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attackcited by 1
  18. 2025One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Modelscited by 1
  19. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Modelscited by 0
  20. 2026A Two-Stage Globally-Diverse Adversarial Attack for Vision-Language Pre-training Modelscited by 0
  21. 2026From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderationcited by 0
  22. 2026Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Modelscited by 0
  23. 2026JECA^2: Judgment-Explanation Consistent Adversarial Attack against Forensic Vision-Language Modelscited by 0
  24. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correctioncited by 0
  25. 2026DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddingscited by 0
  26. 2026On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspacescited by 0
  27. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domaincited by 0
  28. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens
  2. 2024Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks
  3. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  4. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  5. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  6. 2025Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
  7. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models
  8. 2023Adversarial Prompt Tuning for Vision-Language Models