Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
Abstract (arXiv)
Although vision-language pre-training (VLP) models have achieved remarkable progress on cross-modal tasks, they remain vulnerable to adversarial attacks. Using data augmentation and cross-modal interactions to generate transferable adversarial examples on surrogate models, transfer-based black-box attacks have become the mainstream methods in attacking VLP models, as they are more practical in real-world scenarios. However, their transferability may be limited due to the differences on feature representation across different models. To this end, we propose a new attack paradigm called Feedback-based Modal Mutual Search (FMMS). FMMS introduces a novel modal mutual loss (MML), aiming to push away the matched image-text pairs while randomly drawing mismatched pairs closer in feature space, guiding the update directions of the adversarial examples. Additionally, FMMS leverages the target model feedback to iteratively refine adversarial examples, driving them into the adversarial region. To our knowledge, this is the first work to exploit target model feedback to explore multi-modality adversarial boundaries. Extensive empirical evaluations on Flickr30K and MSCOCO datasets for image-text matching tasks show that FMMS significantly outperforms the state-of-the-art baselines.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 40 references that are in the map. First posted on arXiv 27 Aug 2024.
Builds on
36 of 40 references.
Closest ancestors · 5
Same direction, newest first
- 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory19 Mar
- 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
- 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization7 Dec
- 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
- 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun
Other references · 31
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2016Adversarial examples in the physical worldcited by 1,769
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
19 more
- 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
- 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
- 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 462
- 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
- 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
- 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
- 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
- 2018AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networkscited by 137
- 2019Sign-OPT: A Query-Efficient Hard-label Adversarial Attackcited by 120
- 2020QEBA: Query-Efficient Boundary-Based Blackbox Attackcited by 104
- 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
- 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 95
- 2022Parallel Rectangle Flip Attack: A Query-based Black-box Attack against Object Detectioncited by 79
- 2020SurFree: a fast surrogate-free black-box attackcited by 60
- 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMixcited by 45
- 2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 35
- 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanationscited by 24
- 2018Image Captioningcited by 14
- 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
- 2024Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
- 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
- 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
- 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
- 2025MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models
- 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models
- 2024Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models