mlsanityData to 7 Oct 2026

Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models

arXiv 2307.14061 ↗First posted 26 Jul 2023cs.CVcited by 106

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred to attack other black-box models. However, existing work has mainly focused on investigating white-box attacks. In this paper, we present the first study to investigate the adversarial transferability of recent VLP models. We observe that existing methods exhibit much lower transferability, compared to the strong attack performance in white-box settings. The transferability degradation is partly caused by the under-utilization of cross-modal interactions. Particularly, unlike unimodal learning, VLP models rely heavily on cross-modal interactions and the multimodal alignments are many-to-many, e.g., an image can be described in various natural languages. To this end, we propose a highly transferable Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions and incorporates alignment-preserving augmentation with cross-modal guidance. Experimental results demonstrate that SGA could generate adversarial examples that can strongly transfer across different VLP models on multiple downstream vision-language tasks. On image-text retrieval, SGA significantly enhances the attack success rate for transfer attacks from ALBEF to TCL by a large margin (at least 9.78% and up to 30.21%), compared to the state-of-the-art.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 44 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

35 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
22 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  3. 2016Modeling Context in Referring Expressionscited by 1,054
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  7. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  8. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  9. 2021Intriguing Properties of Vision Transformerscited by 446
  10. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  11. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  13. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  14. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  15. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  16. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  17. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
  18. 2021Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translationcited by 41
  19. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  20. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  21. 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9
  22. 2021Dense Contrastive Visual-Linguistic Pretrainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 33; 2025: 37; 2026: 29.

106 papers.

Sort:

By area:

  1. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  2. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  3. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 43
  4. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Modelscited by 43
  5. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
  6. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 39
  7. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
  8. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 35
  9. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Modelscited by 31
  10. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  11. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  12. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
38 more
  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
  2. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  3. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  4. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environmentscited by 22
  5. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Modelscited by 21
  6. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  7. 2025Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Modelscited by 19
  8. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  9. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  10. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learningcited by 16
  11. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  12. 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questionscited by 14
  13. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  14. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  15. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  16. 2024Rethinking Impersonation and Dodging Attacks on Face Recognition Systemscited by 7
  17. 2024Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interactioncited by 7
  18. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approachcited by 7
  19. 2025Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approachcited by 7
  20. 2025QAVA: Query-Agnostic Visual Attack to Large Vision-Language Modelscited by 7
  21. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  22. 2023Data Optimization in Deep Learning: A Surveycited by 6
  23. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  24. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Modelscited by 5
  25. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  26. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  27. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projectorcited by 5
  28. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  29. 2023MVPatch: More Vivid Patch for Adversarial Camouflaged Attacks on Object Detectors in the Physical Worldcited by 4
  30. 2024Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Modelscited by 4
  31. 2024PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Drivingcited by 4
  32. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Modelscited by 4
  33. 2024Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Modelscited by 4
  34. 2024Improving the Transferability of Adversarial Attacks on Face Recognition with Diverse Parameters Augmentationcited by 3
  35. 2025CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMscited by 3
  36. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 3
  37. 2025Poisoning Prompt-Guided Sampling in Video Large Language Modelscited by 3
  38. 2024Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Modelscited by 2

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models
  2. 2026Breaking the weakest link to evade vision language models
  3. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  4. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens
  5. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
  6. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models
  7. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  8. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation