mlsanityData to 7 Oct 2026

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

arXiv 2403.10883 ↗First posted 16 Mar 2024cs.CV cs.CR cs.MMcited by 7

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging modality interaction through embedding guidance and interaction enhancement. Specifically, attacking text at the embedding level while preserving semantics, as well as utilizing interaction image gradients to enhance constraints on perturbations of texts and images. Significantly, in the image-text retrieval task on Flickr30K dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL, CLIPViT and CLIPCNN by 8.11%-16.75% over state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior performance in cross-task generalization scenarios. Our work addresses the underexplored realm of transfer attacks on VLP models, shedding light on the importance of modality interaction for enhanced adversarial robustness.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 38 references that are in the map. First posted on arXiv 16 Mar 2024.

Builds on

27 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2016Adversarial Machine Learning at Scalecited by 934
13 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  2. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  3. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  4. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  5. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  6. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  7. 2023Content-based Unrestricted Adversarial Attackcited by 73
  8. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  9. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  10. 2023Query-Efficient Decision-based Black-Box Patch Attackcited by 19
  11. 2022Disentangled Multimodal Representation Learning for Recommendationcited by 15
  12. 2020Fine-Grained Image Captioning with Global-Local Discriminative Objectivecited by 12
  13. 2023Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignmentcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 3.

7 papers.

  1. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  2. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  3. 2024PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Drivingcited by 4
  4. 2026VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijackingcited by 3
  5. 2025Proxy-Embedding as an Adversarial Teacher: An Embedding-Guided Bidirectional Attack for Referring Expression Segmentation Modelscited by 0
  6. 2026JECA^2: Judgment-Explanation Consistent Adversarial Attack against Forensic Vision-Language Modelscited by 0
  7. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
  2. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  3. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  4. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
  5. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
  6. 2026Breaking the weakest link to evade vision language models
  7. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
  8. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective