# Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective

arXiv: https://arxiv.org/abs/2404.19287 · HTML: https://mlsanity.com/paper/2404.19287/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 62 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [On Evaluating Adversarial Robustness of Large Vision-Language Models](https://mlsanity.com/paper/2305.16934/) (2023)
- [Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients](https://mlsanity.com/paper/1711.09404/) (2017)
- [A study of the effect of JPG compression on adversarial images](https://mlsanity.com/paper/1608.00853/) (2016)
- [Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning](https://mlsanity.com/paper/2003.12862/) (2020)
- [Towards Deep Neural Network Architectures Robust to Adversarial Examples](https://mlsanity.com/paper/1412.5068/) (2014)
- [Towards Adversarial Attack on Vision-Language Pre-training Models](https://mlsanity.com/paper/2206.09391/) (2022)
- [Understanding Zero-Shot Adversarial Robustness for Large-Scale Models](https://mlsanity.com/paper/2212.07016/) (2022)
- [Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models](https://mlsanity.com/paper/2307.14061/) (2023)
- [PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization](https://mlsanity.com/paper/2307.15199/) (2023)

## Built on by (most cited first, 28 of 28)

- [DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models](https://mlsanity.com/paper/2411.00836/) (2024)
- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization](https://mlsanity.com/paper/2505.16640/) (2025)
- [TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models](https://mlsanity.com/paper/2411.13136/) (2024)
- [Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs](https://mlsanity.com/paper/2601.19202/) (2026)
- [Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Models](https://mlsanity.com/paper/2509.19994/) (2025)
- [Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models](https://mlsanity.com/paper/2407.02716/) (2024)
- [Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models](https://mlsanity.com/paper/2411.05056/) (2024)
- [Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP](https://mlsanity.com/paper/2601.19210/) (2026)
- [Are Video Reasoning Models Ready to Go Outside?](https://mlsanity.com/paper/2603.10652/) (2026)
- [On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabeling](https://mlsanity.com/paper/2506.21874/) (2025)
- [ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP](https://mlsanity.com/paper/2511.17362/) (2025)
- [Adversarial Defense in Vision-Language Models: An Overview](https://mlsanity.com/paper/2601.12443/) (2026)
- [Hierarchical Attacks for Multi-Modal Multi-Agent Reasoning](https://mlsanity.com/paper/2605.13213/) (2026)
- [Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration](https://mlsanity.com/paper/2505.11895/) (2025)
- [Dual-Path Stable Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2505.18770/) (2025)
- [LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders](https://mlsanity.com/paper/2505.18884/) (2025)
- [Coordinated Robustness Evaluation Framework for Vision-Language Models](https://mlsanity.com/paper/2506.05429/) (2025)
- [Evaluating Robustness of Vision-Language Models Under Noisy Conditions](https://mlsanity.com/paper/2509.12492/) (2025)
- [RLBind: Adversarial-Invariant Cross-Modal Alignment for Unified Robust Embeddings](https://mlsanity.com/paper/2509.14383/) (2025)
- [ORCA: An Agentic Reasoning Framework for Hallucination and Adversarial Robustness in Vision-Language Models](https://mlsanity.com/paper/2509.15435/) (2025)
- [Adversarial Robustness in Zero-Shot Learning:An Empirical Study on Class and Concept-Level Vulnerabilities](https://mlsanity.com/paper/2512.18651/) (2025)
- [On the Adversarial Robustness of 3D Large Vision-Language Models](https://mlsanity.com/paper/2601.06464/) (2026)
- [NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving](https://mlsanity.com/paper/2602.13293/) (2026)
- [TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models](https://mlsanity.com/paper/2605.17577/) (2026)
- [JECA^2: Judgment-Explanation Consistent Adversarial Attack against Forensic Vision-Language Models](https://mlsanity.com/paper/2605.28609/) (2026)
- [Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization](https://mlsanity.com/paper/2608.29924/) (2026)

## Abstract (arXiv)

Pretrained vision-language models (VLMs) like CLIP exhibit exceptional generalization across diverse downstream tasks. While recent studies reveal their vulnerability to adversarial attacks, research to date has primarily focused on enhancing the robustness of image encoders against image-based attacks, with defenses against text-based and multimodal attacks remaining largely unexplored. To this end, this work presents the first comprehensive study on improving the adversarial robustness of VLMs against attacks targeting image, text, and multimodal inputs. This is achieved by proposing multimodal contrastive adversarial training (MMCoA). Such an approach strengthens the robustness of both image and text encoders by aligning the clean text embeddings with adversarial image embeddings, and adversarial text embeddings with clean image embeddings. The robustness of the proposed MMCoA is examined against existing defense methods over image, text, and multimodal attacks on the CLIP model. Extensive experiments on 15 datasets across two tasks reveal the characteristics of different adversarial defense methods under distinct distribution shifts and dataset complexities across the three attack types. This paves the way for a unified framework of adversarial robustness against different modality attacks, opening up new possibilities for securing VLMs against multimodal attacks. The code is available at https://github.com/ElleZWQ/MMCoA.git.