mlsanityData to 13 Oct 2026

Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

arXiv 2404.19287 ↗First posted 30 Apr 2024cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Pretrained vision-language models (VLMs) like CLIP exhibit exceptional generalization across diverse downstream tasks. While recent studies reveal their vulnerability to adversarial attacks, research to date has primarily focused on enhancing the robustness of image encoders against image-based attacks, with defenses against text-based and multimodal attacks remaining largely unexplored. To this end, this work presents the first comprehensive study on improving the adversarial robustness of VLMs against attacks targeting image, text, and multimodal inputs. This is achieved by proposing multimodal contrastive adversarial training (MMCoA). Such an approach strengthens the robustness of both image and text encoders by aligning the clean text embeddings with adversarial image embeddings, and adversarial text embeddings with clean image embeddings. The robustness of the proposed MMCoA is examined against existing defense methods over image, text, and multimodal attacks on the CLIP model. Extensive experiments on 15 datasets across two tasks reveal the characteristics of different adversarial defense methods under distinct distribution shifts and dataset complexities across the three attack types. This paves the way for a unified framework of adversarial robustness against different modality attacks, opening up new possibilities for securing VLMs against multimodal attacks. The code is available at https://github.com/ElleZWQ/MMCoA.git.

Key facts. Cited by 28 papers in the map (data through 13 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 62 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

39 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  2. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  3. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  5. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  6. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  5. 2013Intriguing properties of neural networkscited by 5,137
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,022
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,276
  9. 2013Describing Textures in the Wildcited by 2,115
  10. 2016Adversarial examples in the physical worldcited by 1,767
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,709
  12. 2022Visual Prompt Tuningcited by 1,677
21 more
  1. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,316
  2. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,023
  3. 2021Robust fine-tuning of zero-shot modelscited by 790
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  5. 2017Countering Adversarial Images using Input Transformationscited by 519
  6. 2021Image Segmentation Using Text and Image Promptscited by 511
  7. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 490
  8. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  9. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 381
  10. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  11. 2016A study of the effect of JPG compression on adversarial imagescited by 197
  12. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  13. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  14. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  15. 2023Prompt-based Distribution Alignment for Unsupervised Domain Adaptationcited by 44
  16. 2017Enhancing Robustness of Machine Learning Systems via Data Transformationscited by 38
  17. 2023Randomized Adversarial Training via Taylor Expansioncited by 28
  18. 2017Regularizing deep networks using efficient layerwise adversarial trainingcited by 24
  19. 2021Towards Robust Neural Image Compression: Adversarial Attack and Model Finetuningcited by 24
  20. 2022Dynamics-aware Adversarial Attack of Adaptive Neural Networkscited by 5
  21. 2023Advancing Adversarial Training by Injecting Booster Signalcited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 10.

28 papers.

Sort:

By area:

  1. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 146
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 64
  3. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 60
  4. 2025BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimizationcited by 38
  5. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 23
  6. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 7
  7. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  8. 2024Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Modelscited by 4
  9. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  10. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  11. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  12. 2025On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabelingcited by 1
16 more
  1. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1
  2. 2026Adversarial Defense in Vision-Language Models: An Overviewcited by 1
  3. 2026Hierarchical Attacks for Multi-Modal Multi-Agent Reasoningcited by 1
  4. 2025Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibrationcited by 0
  5. 2025Dual-Path Stable Soft Prompt Generation for Domain Generalizationcited by 0
  6. 2025LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoderscited by 0
  7. 2025Coordinated Robustness Evaluation Framework for Vision-Language Modelscited by 0
  8. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditionscited by 0
  9. 2025RLBind: Adversarial-Invariant Cross-Modal Alignment for Unified Robust Embeddingscited by 0
  10. 2025ORCA: An Agentic Reasoning Framework for Hallucination and Adversarial Robustness in Vision-Language Modelscited by 0
  11. 2025Adversarial Robustness in Zero-Shot Learning:An Empirical Study on Class and Concept-Level Vulnerabilitiescited by 0
  12. 2026On the Adversarial Robustness of 3D Large Vision-Language Modelscited by 0
  13. 2026NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Drivingcited by 0
  14. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  15. 2026JECA^2: Judgment-Explanation Consistent Adversarial Attack against Forensic Vision-Language Modelscited by 0
  16. 2026Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models
  2. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  3. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  4. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  5. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  6. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
  7. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
  8. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness