mlsanityData to 14 Oct 2026

One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models

arXiv 2403.01849 ↗First posted 4 Mar 2024cs.CV cs.AI cs.LGcited by 48

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Large pre-trained Vision-Language Models (VLMs) like CLIP, despite having remarkable generalization ability, are highly vulnerable to adversarial examples. This work studies the adversarial robustness of VLMs from the novel perspective of the text prompt instead of the extensively studied model weights (frozen in this work). We first show that the effectiveness of both adversarial attack and defense are sensitive to the used text prompt. Inspired by this, we propose a method to improve resilience to adversarial attacks by learning a robust text prompt for VLMs. The proposed method, named Adversarial Prompt Tuning (APT), is effective while being both computationally and data efficient. Extensive experiments are conducted across 15 datasets and 4 data sparsity schemes (from 1-shot to full training data settings) to show APT's superiority over hand-engineered prompts and other state-of-the-art adaption methods. APT demonstrated excellent abilities in terms of the in-distribution performance and the generalization under input distribution shift and across datasets. Surprisingly, by simply adding one learned word to the prompts, APT can significantly boost the accuracy and robustness (epsilon=4/255) over the hand-engineered prompts by +13% and +8.5% on average respectively. The improvement further increases, in our most effective setting, to +26.4% for accuracy and +16.7% for robustness. Code is available at https://github.com/TreeLLi/APT.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 77 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

59 of 77 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  2. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
38 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  8. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  10. 2014How transferable are features in deep neural networks?cited by 1,225
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  12. 2018Robustness May Be at Odds with Accuracycited by 880
  13. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  14. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  15. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  16. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  17. 2020Overfitting in adversarially robust deep learningcited by 506
  18. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  19. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  20. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  21. 2017Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?cited by 221
  22. 2020Understanding and Improving Fast Adversarial Trainingcited by 214
  23. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 208
  24. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  25. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 153
  26. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  27. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  28. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  29. 2020Adversarial Self-Supervised Contrastive Learningcited by 139
  30. 2022Neural Prompt Searchcited by 135
  31. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 85
  32. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  33. 2022Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Trainingcited by 45
  34. 2022Make Some Noise: Reliable and Efficient Single-Step Adversarial Trainingcited by 43
  35. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  36. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  37. 2022Visual Prompting for Adversarial Robustnesscited by 34
  38. 2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classificationcited by 34

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 21; 2026: 16.

48 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  2. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  3. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  4. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  5. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 23
  6. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  7. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  8. 2024Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attackcited by 18
  9. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
  10. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 13
  11. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  12. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Modelscited by 11
36 more
  1. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
  2. 2025Emerging Cyber Attack Risks of Medical AI Agentscited by 8
  3. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  4. 2024Adversarial Training: A Surveycited by 6
  5. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  6. 2025Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Modelscited by 6
  7. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  8. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
  9. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 3
  10. 2025Processing and acquisition traces in visual encoders: What does CLIP know about your camera?cited by 3
  11. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  12. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  13. 2025Hydra: An Agentic Reasoning Approach for Enhancing Adversarial Robustness and Mitigating Hallucinations in Vision-Language Modelscited by 2
  14. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  15. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  16. 2025Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIPcited by 1
  17. 2025FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Modelscited by 1
  18. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1
  19. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  20. 2026Adversarial Defense in Vision-Language Models: An Overviewcited by 1
  21. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Modelscited by 1
  22. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
  23. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  24. 2026Hierarchically Robust Zero-shot Vision-language Modelscited by 1
  25. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  26. 2026Semantic Robustness Certification for Vision-Language Modelscited by 1
  27. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  28. 2025ORCA: An Agentic Reasoning Framework for Hallucination and Adversarial Robustness in Vision-Language Modelscited by 0
  29. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  30. 2026AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Modelscited by 0
  31. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  32. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Modelscited by 0
  33. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 0
  34. 2026T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Modelscited by 0
  35. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0
  36. 2026Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adversarial Prompt Tuning for Vision-Language Models
  2. 2024Revisiting the Robust Generalization of Adversarial Prompt Tuning
  3. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  4. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models
  5. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  6. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  7. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  8. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models