mlsanityData to 7 Oct 2026

SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation

arXiv 2312.04913 ↗First posted 8 Dec 2023cs.CV cs.AI cs.CRcited by 39

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Current Visual-Language Pre-training (VLP) models are vulnerable to adversarial examples. These adversarial examples present substantial security risks to VLP models, as they can leverage inherent weaknesses in the models, resulting in incorrect predictions. In contrast to white-box adversarial attacks, transfer attacks (where the adversary crafts adversarial examples on a white-box model to fool another black-box model) are more reflective of real-world scenarios, thus making them more meaningful for research. By summarizing and analyzing existing research, we identified two factors that can influence the efficacy of transfer attacks on VLP models: inter-modal interaction and data diversity. Based on these insights, we propose a self-augment-based transfer attack method, termed SA-Attack. Specifically, during the generation of adversarial images and adversarial texts, we apply different data augmentation methods to the image modality and text modality, respectively, with the aim of improving the adversarial transferability of the generated adversarial images and texts. Experiments conducted on the FLickr30K and COCO datasets have validated the effectiveness of our method. Our code will be available after this paper is accepted.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 87 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

74 of 87 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 72

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
38 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2016Adversarial Machine Learning at Scalecited by 934
  3. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  4. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  5. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  6. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  8. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  9. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  12. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 242
  13. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  14. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  15. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  16. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  17. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 165
  18. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  19. 2021Admix: Enhancing the Transferability of Adversarial Attackscited by 155
  20. 2019Population Based Augmentation: Efficient Learning of Augmentation Policy Schedulescited by 153
  21. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  22. 2021Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognitioncited by 95
  23. 2019SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editingcited by 93
  24. 2022Improving Adversarial Transferability via Neuron Attribution-Based Attackscited by 92
  25. 2021Improving Transferability of Adversarial Patches on Face Recognition with Generative Modelscited by 80
  26. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  27. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  28. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  29. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 67
  30. 2023Diffusion-Based Adversarial Sample Generation for Improved Stealthiness and Controllabilitycited by 66
  31. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  32. 2023Structure Invariant Transformation for better Adversarial Transferabilitycited by 61
  33. 2022T-SEA: Transfer-based Self-Ensemble Attack on Object Detectioncited by 59
  34. 2022Improving the Transferability of Targeted Adversarial Examples through Object-Based Diverse Inputcited by 49
  35. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 43
  36. 2022Adv-Attribute: Inconspicuous and Transferable Adversarial Attack on Face Recognitioncited by 42
  37. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  38. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Modelcited by 41

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 10; 2026: 8.

39 papers.

Sort:

By area:

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  2. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  5. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 35
  6. 2024Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearningcited by 31
  7. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  8. 2024Compromising Embodied Agents with Contextual Backdoor Attackscited by 29
  9. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  10. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  11. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  12. 2024LanEvil: Benchmarking the Robustness of Lane Detection to Environmental Illusionscited by 18
27 more
  1. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  2. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  3. 2024Object Detectors in the Open Environment: Challenges, Solutions, and Outlookcited by 13
  4. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  5. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approachcited by 7
  6. 2024Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threatscited by 6
  7. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  8. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Modelscited by 5
  9. 2024Towards Robust Object Detection: Identifying and Removing Backdoors via Module Inconsistency Analysiscited by 4
  10. 2024The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models?cited by 2
  11. 2025MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Modelscited by 2
  12. 2025Attacking Attention of Foundation Models Disrupts Downstream Taskscited by 2
  13. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  14. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  15. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1
  16. 2025Adversarial Video Promotion Against Text-to-Video Retrievalcited by 1
  17. 2026Multimodal Backdoor Attack on VLMs for Autonomous Driving via Graffiti and Cross-Lingual Triggerscited by 1
  18. 2026Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspectivecited by 1
  19. 2024Partially Recentralization Softmax Loss for Vision-Language Models Robustnesscited by 0
  20. 2024Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Modelscited by 0
  21. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
  22. 2025Pay Less Attention to Function Words for Free Robustness of Vision-Language Modelscited by 0
  23. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interactioncited by 0
  24. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  25. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0
  26. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipelinecited by 0
  27. 2026CIVA: Critic-Induced Value-Subspace Attacks on Visual World-Model Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization
  2. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack
  3. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  4. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  5. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  6. 2024Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
  7. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models
  8. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models