mlsanityData to 14 Oct 2026

On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline

arXiv 2607.14974 ↗First posted 16 Jul 2026cs.CV cs.CRcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Pre-training Models (VLPMs) are known to be vulnerable to adversarial attacks. Recent transferable attacks on VLPMs have followed a common pipeline with complicated loss functions or multi-stage text/image attacks. However, in this paper, we demonstrate that such a sophisticated attack pipeline can be simpler yet more successful. Specifically, we identify three previously overlooked issues caused by inappropriate cross-modal interactions and excessive operations. To address them, we propose the Simple Vision-Language Attack (SimVLA) pipeline, which observably improves transferability and efficiency. Experiments on four datasets and three downstream tasks validate the superiority of our pipeline. For instance, on Flickr30k text-image retrieval dataset, our SimVLA outperforms the SOTA baseline in R@1 transferability by 8.01%-14.71%, while consuming only about 35.73% of the time and 46.26% of the max VRAM. Overall, the superiority of our SimVLA highlights the importance of leveraging domain knowledge (e.g., our proposed cross-modal word identification), while blindly pursuing intricate operations (e.g, complex loss functions and redundant multi-stage designs) may even be harmful. We hope our SimVLA can serve as a simple yet effective backbone for future extensions. Code is available at https://github.com/RYC-98/SimVLA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 47 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

40 of 47 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments28 Nov
  2. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack4 Nov
  3. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models25 Jul
  4. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  5. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory19 Mar
  6. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  7. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  8. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
  9. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning24 Aug
  10. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  11. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  12. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2016Modeling Context in Referring Expressionscited by 1,063
16 more
  1. 2017Countering Adversarial Images using Input Transformationscited by 520
  2. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  3. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
  4. 2020Circle Loss: A Unified Perspective of Pair Similarity Optimizationcited by 403
  5. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  6. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
  7. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  8. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 130
  9. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  10. 2020On Success and Simplicity: A Second Look at Transferable Targeted Attackscited by 89
  11. 2021Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferabilitycited by 84
  12. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 68
  13. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  14. 2021A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attackscited by 34
  15. 2024Improving Integrated Gradient-based Transferable Adversarial Examples by Refining the Integration Pathcited by 10
  16. 2025Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinementcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  2. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
  3. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
  4. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  5. 2024Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
  6. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
  7. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  8. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens