mlsanityData to 14 Oct 2026

Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning

arXiv 2308.12636 ↗First posted 24 Aug 2023cs.MMcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

The integration of visual and textual data in Vision-Language Pre-training (VLP) models is crucial for enhancing vision-language understanding. However, the adversarial robustness of these models, especially in the alignment of image-text features, has not yet been sufficiently explored. In this paper, we introduce a novel gradient-based multimodal adversarial attack method, underpinned by contrastive learning, to improve the transferability of multimodal adversarial samples in VLP models. This method concurrently generates adversarial texts and images within imperceptive perturbation, employing both image-text and intra-modal contrastive loss. We evaluate the effectiveness of our approach on image-text retrieval and visual entailment tasks, using publicly available datasets in a black-box setting. Extensive experiments indicate a significant advancement over existing single-modal transfer-based adversarial attack methods and current multimodal adversarial attack approaches.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 73 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

54 of 73 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  3. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  10. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  11. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  12. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  13. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  14. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
  15. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  16. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
  17. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  18. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  19. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
  20. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  21. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  22. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  23. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  24. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 167
  25. 2021Feature Importance-aware Transferable Adversarial Attackscited by 127
  26. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  27. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  28. 2020Backpropagating Linearly Improves Transferability of Adversarial Examplescited by 79
  29. 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78
  30. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 68
  31. 2021On Generating Transferable Targeted Perturbationscited by 65
  32. 2019Black-box Adversarial Attacks on Video Recognition Modelscited by 58
  33. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  34. 2022Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Priorcited by 38
  35. 2022Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspectivecited by 37
  36. 2021Boosting Transferability of Targeted Adversarial Examples via Hierarchical Generative Networkscited by 32
  37. 2023TransFool: An Adversarial Attack against Neural Machine Translation Modelscited by 12
  38. 2023Boosting Adversarial Transferability with Learnable Patch-wise Maskscited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 6; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  2. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  3. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Modelscited by 21
  4. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  5. 2024Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Modelscited by 19
  6. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Modelscited by 9
  7. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  8. 2026Information-Theoretic Constraints for Continual Vision-Language-Action Alignmentcited by 3
  9. 2025Attacking Attention of Foundation Models Disrupts Downstream Taskscited by 2
  10. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  11. 2025B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Groundingcited by 0
  12. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interactioncited by 0
4 more
  1. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  2. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipelinecited by 0
  3. 2026Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Modelcited by 0
  4. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  2. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
  3. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  4. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory
  5. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models
  6. 2025Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack
  7. 2024Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
  8. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack