Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attack
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
Abstract (arXiv)
Multimodal Large Language Models (MLLMs), built upon LLMs, have recently gained attention for their capabilities in image recognition and understanding. However, while MLLMs are vulnerable to adversarial attacks, the transferability of these attacks across different models remains limited, especially under targeted attack setting. Existing methods primarily focus on vision-specific perturbations but struggle with the complex nature of vision-language modality alignment. In this work, we introduce the Dynamic Vision-Language Alignment (DynVLA) Attack, a novel approach that injects dynamic perturbations into the vision-language connector to enhance generalization across diverse vision-language alignment of different models. Our experimental results show that DynVLA significantly improves the transferability of adversarial examples across various MLLMs, including BLIP2, InstructBLIP, MiniGPT4, LLaVA, and closed-source models such as Gemini.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 56 references that are in the map. First posted on arXiv 27 Feb 2025.
Builds on
39 of 56 references.
Closest ancestors · 7
Same direction, newest first
- 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models30 May
- 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models14 Mar
- 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images20 Jan
- 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models4 Dec
- 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
- 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
- 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
Other references · 32
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Visual Instruction Tuningcited by 7,869
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2013Intriguing properties of neural networkscited by 5,155
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
20 more
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
- 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
- 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
- 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
- 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
- 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
- 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
- 2017The Space of Transferable Adversarial Examplescited by 158
- 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 130
- 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
- 2023Structure Invariant Transformation for better Adversarial Transferabilitycited by 62
- 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 44
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Breaking the weakest link to evade vision language models
- 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
- 2026Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs
- 2026Investigating Adversarial Robustness of Multi-modal Large Language Models
- 2025Survey of Adversarial Robustness in Multimodal Large Language Models
- 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
- 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
- 2025Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approach