mlsanityData to 7 Oct 2026

OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization

arXiv 2312.04403 ↗First posted 7 Dec 2023cs.CVcited by 36

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language pre-training (VLP) models demonstrate impressive abilities in processing both images and text. However, they are vulnerable to multi-modal adversarial examples (AEs). Investigating the generation of high-transferability adversarial examples is crucial for uncovering VLP models' vulnerabilities in practical scenarios. Recent works have indicated that leveraging data augmentation and image-text modal interactions can enhance the transferability of adversarial examples for VLP models significantly. However, they do not consider the optimal alignment problem between dataaugmented image-text pairs. This oversight leads to adversarial examples that are overly tailored to the source model, thus limiting improvements in transferability. In our research, we first explore the interplay between image sets produced through data augmentation and their corresponding text sets. We find that augmented image samples can align optimally with certain texts while exhibiting less relevance to others. Motivated by this, we propose an Optimal Transport-based Adversarial Attack, dubbed OT-Attack. The proposed method formulates the features of image and text sets as two distinct distributions and employs optimal transport theory to determine the most efficient mapping between them. This optimal mapping informs our generation of adversarial examples to effectively counteract the overfitting issues. Extensive experiments across various network architectures and datasets in image-text matching tasks reveal that our OT-Attack outperforms existing state-of-the-art methods in terms of adversarial transferability.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 58 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

47 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
33 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  3. 2018Computational Optimal Transportcited by 765
  4. 2020A Survey on Visual Transformercited by 653
  5. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  6. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  7. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  8. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  9. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  12. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  13. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  14. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  15. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  16. 2022Frequency Domain Model Augmentation for Adversarial Attackcited by 128
  17. 2019Gromov-Wasserstein Learning for Graph Matching and Node Embeddingcited by 125
  18. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  19. 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 94
  20. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 67
  21. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  22. 2020Adv-watermark: A Novel Watermark Perturbation for Adversarial Examplescited by 56
  23. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  24. 2022Improving the Transferability of Targeted Adversarial Examples through Object-Based Diverse Inputcited by 49
  25. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  26. 2023A Survey on Transferability of Adversarial Examples across Deep Neural Networkscited by 43
  27. 2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
  28. 2022Prior-Guided Adversarial Initialization for Fast Adversarial Trainingcited by 34
  29. 2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 34
  30. 2021Towards Interpretable Deep Metric Learning with Structural Matchingcited by 30
  31. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanationscited by 24
  32. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  33. 2021Closer Look at the Transferability of Adversarial Examples: How They Fool Different Models Differentlycited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 11; 2026: 7.

36 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  5. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  6. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  7. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  8. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  9. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectorycited by 35
  10. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  11. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  12. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
24 more
  1. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  2. 2025Understanding and Rectifying Safety Perception Distortion in VLMscited by 17
  3. 2024Texture Re-scalable Universal Adversarial Perturbationcited by 9
  4. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approachcited by 7
  5. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  6. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  7. 2025Exploring Adversarial Transferability between Kolmogorov-arnold Networkscited by 4
  8. 2025Understanding Knowledge Transferability for Transfer Learning: A Surveycited by 4
  9. 2025PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systemscited by 4
  10. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
  11. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
  12. 2024The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models?cited by 2
  13. 2025NoPain: No-box Point Cloud Attack via Optimal Transport Singular Boundarycited by 2
  14. 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2
  15. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationshipscited by 1
  16. 2025Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Modelscited by 1
  17. 2025One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Modelscited by 1
  18. 2025Cowpox: Towards the Immunity of VLM-based Multi-Agent Systemscited by 1
  19. 2024Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Modelscited by 0
  20. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interactioncited by 0
  21. 2026MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Modelscited by 0
  22. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0
  23. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipelinecited by 0
  24. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  4. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
  5. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models
  6. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens
  7. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models
  8. 2026Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction