mlsanityData to 7 Oct 2026

Few-Shot Adversarial Prompt Learning on Vision-Language Models

arXiv 2403.14774 ↗First posted 21 Mar 2024cs.CV cs.CL cs.CRcited by 39

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Inspired by the success of vision-language foundation models, previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. However, in practice, they are still unsatisfactory due to several issues, including heavy adaptation cost, suboptimal text supervision, and uncontrolled natural generalization capacity. In this paper, to address these issues, we propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement. Specifically, we achieve this by providing adversarially correlated text supervision that is end-to-end learned from adversarial examples. We also propose a novel training objective that enhances the consistency of multi-modal features while encourages differentiated uni-modal features between natural and adversarial examples. The proposed framework gives access to learn adversarial text supervision, which provides superior cross-modal adversarial alignment and matches state-of-the-art zero-shot adversarial robustness with only 1% training data. Code is available at: https://github.com/lionel-w2/FAP.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 91 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

69 of 91 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  2. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  4. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  5. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2022Visual Prompt Tuningcited by 1,674
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2021Grounded Language-Image Pre-trainingcited by 1,147
  14. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2021GPT Understands, Toocited by 715
  17. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  18. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  19. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  20. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  21. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  22. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  23. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  24. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  25. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  26. 2022Visual Prompting via Image Inpaintingcited by 248
  27. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  28. 2022Prompt Distribution Learningcited by 235
  29. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  30. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  31. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  32. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  33. 2021When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?cited by 76
  34. 2022Understanding Robust Overfitting of Adversarial Training and Beyondcited by 50
  35. 2022Harnessing Out-Of-Distribution Examples via Augmenting Content and Stylecited by 43
  36. 2022Self-Ensemble Adversarial Training for Improved Robustnesscited by 38
  37. 2021CausalAdv: Adversarial Robustness through the Lens of Causalitycited by 37
  38. 2021On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learningcited by 29

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 23; 2026: 12.

39 papers.

Sort:

By area:

  1. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  2. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  3. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  4. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  5. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  6. 2025Continual Multimodal Contrastive Learningcited by 19
  7. 2025Principled Multimodal Representation Learningcited by 19
  8. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  9. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  10. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  11. 2025Where, What, Why: Towards Explainable Driver Attention Predictioncited by 10
  12. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
27 more
  1. 2025Logic Unseen: Revealing the Logical Blindspots of Vision-Language Modelscited by 7
  2. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  3. 2025Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Modelscited by 6
  4. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
  5. 2025VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuningcited by 3
  6. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 3
  7. 2025Enhancing Target-unspecific Tasks through a Features Matrixcited by 2
  8. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 2
  9. 2025UtilGen: Utility-Centric Generative Data Augmentation with Dual-Level Task Adaptationcited by 2
  10. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  11. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  12. 2025Adapting Vision-Language Models for Evaluating World Modelscited by 1
  13. 2025FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Modelscited by 1
  14. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  15. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  16. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  17. 2025Provably Robust Adaptation for Language-Empowered Foundation Modelscited by 0
  18. 2025Automatic Attack Discovery for Few-Shot Class-Incremental Learning via Large Language Modelscited by 0
  19. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  20. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
  21. 2026Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labelscited by 0
  22. 2026AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Modelscited by 0
  23. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  24. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Modelscited by 0
  25. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 0
  26. 2026T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Modelscited by 0
  27. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Robust Generalization of Adversarial Prompt Tuning
  2. 2026Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
  3. 2026Complementary Text-Guided Attention for Zero-Shot Adversarial Robustness
  4. 2026Hierarchically Robust Zero-shot Vision-language Models
  5. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models
  6. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustness
  7. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models
  8. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models