mlsanityData to 14 Oct 2026

Adversarial Prompt Tuning for Vision-Language Models

arXiv 2311.11261 ↗First posted 19 Nov 2023cs.CV cs.AIcited by 44

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

With the rapid advancement of multimodal learning, pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capacities in bridging the gap between visual and language modalities. However, these models remain vulnerable to adversarial attacks, particularly in the image modality, presenting considerable security risks. This paper introduces Adversarial Prompt Tuning (AdvPT), a novel technique to enhance the adversarial robustness of image encoders in VLMs. AdvPT innovatively leverages learnable text prompts and aligns them with adversarial image embeddings, to address the vulnerabilities inherent in VLMs without the need for extensive parameter training or modification of the model architecture. We demonstrate that AdvPT improves resistance against white-box and black-box adversarial attacks and exhibits a synergistic effect when combined with existing image-processing-based defense techniques, further boosting defensive capabilities. Comprehensive experimental analyses provide insights into adversarial prompt tuning, a novel paradigm devoted to improving resistance to adversarial images through textual input modifications, paving the way for future robust multimodal learning research. These findings open up new possibilities for enhancing the security of VLMs. Our code is available at https://github.com/jiamingzhang94/Adversarial-Prompt-Tuning.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 50 references that are in the map. First posted on arXiv 19 Nov 2023.

Builds on

40 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  2. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  3. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2013Describing Textures in the Wildcited by 2,125
25 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2022Visual Prompt Tuningcited by 1,681
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  8. 2019Natural Adversarial Examplescited by 1,408
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2021Learning to Prompt for Continual Learningcited by 944
  11. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  13. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  14. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  15. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  16. 2017Countering Adversarial Images using Input Transformationscited by 520
  17. 2022Diffusion Models for Adversarial Purificationcited by 476
  18. 2017Mitigating Adversarial Effects Through Randomizationcited by 383
  19. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  20. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
  21. 2021Adversarial purification with Score-based generative modelscited by 134
  22. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  23. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 68
  24. 2019Image Super-Resolution as a Defense Against Adversarial Attackscited by 65
  25. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 12; 2026: 20.

44 papers.

Sort:

By area:

  1. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 84
  2. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  3. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 33
  4. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
  5. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 23
  6. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  7. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 19
  8. 2025A Survey on Efficient Vision-Language Modelscited by 19
  9. 2024Active Learning for Vision-Language Modelscited by 15
  10. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
  11. 2024Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuningcited by 12
  12. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
32 more
  1. 2024AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuningcited by 7
  2. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  3. 2023Asymmetric Bias in Text-to-Image Generation with Adversarial Attackscited by 5
  4. 2024AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimizationcited by 5
  5. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  6. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  7. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attackscited by 3
  8. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 3
  9. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 3
  10. 2025VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Servicecited by 3
  11. 2025HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learningcited by 3
  12. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  13. 2025FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Modelscited by 1
  14. 2026Continual Learning with Vision-Language Models via Semantic-Geometry Preservationcited by 1
  15. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Modelscited by 1
  16. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
  17. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  18. 2026Hierarchically Robust Zero-shot Vision-language Modelscited by 1
  19. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  20. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  21. 2026Feature-Space Smoothing: Certified Robustness of Deep Representationscited by 0
  22. 2026Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluationcited by 0
  23. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  24. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0
  25. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
  26. 2026STARE: Step-wise Temporal Alignment and Red-teaming Engine for Multi-modal Toxicity Attackcited by 0
  27. 2026From Clouds to Hallucinations: Atmospheric Retrieval Hijacking in Remote Sensing Vision-Language RAGcited by 0
  28. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  29. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Modelscited by 0
  30. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 0
  31. 2026T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Modelscited by 0
  32. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  2. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  3. 2024Revisiting the Robust Generalization of Adversarial Prompt Tuning
  4. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors
  5. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  6. 2026Answer It or Not? Towards Prompt-Adapted Adversarial Images on Vision Language Models
  7. 2026Breaking the weakest link to evade vision language models
  8. 2026Adversarial Defense in Vision-Language Models: An Overview