mlsanityData to 7 Oct 2026

TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models

arXiv 2411.13136 ↗First posted 20 Nov 2024cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Large pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated excellent zero-shot generalizability across various downstream tasks. However, recent studies have shown that the inference performance of CLIP can be greatly degraded by small adversarial perturbations, especially its visual modality, posing significant safety threats. To mitigate this vulnerability, in this paper, we propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. Specifically, it is an unsupervised method that optimizes the defensive prompts for each test sample by minimizing a multi-view entropy and aligning adversarial-clean distributions. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets, demonstrating that it enhances the zero-shot adversarial robustness of the original CLIP by at least 48.9% against AutoAttack (AA), while largely maintaining performance on clean examples. Moreover, TAPT outperforms existing adversarial prompt tuning methods across various backbones, achieving an average robustness improvement of at least 36.6%.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 69 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

54 of 69 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  2. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  3. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  4. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  5. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  6. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  7. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  8. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  9. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  10. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  11. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  12. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
1 more
  1. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
29 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  7. 2022Continual Test-Time Domain Adaptationcited by 520
  8. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  9. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  11. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  12. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  13. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  14. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 235
  15. 2018Disentangling Adversarial Robustness and Generalizationcited by 171
  16. 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 156
  17. 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
  18. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  19. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 100
  20. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  21. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  22. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  23. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  24. 2024Revisiting Adversarial Training at Scalecited by 33
  25. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learningcited by 16
  26. 2023Multi-event Video-Text Retrievalcited by 15
  27. 2020Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustnesscited by 11
  28. 2023Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Linecited by 11
  29. 2024AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 13.

22 papers.

Sort:

By area:

  1. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  2. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  3. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 3
  4. 2026FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustnesscited by 3
  5. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  6. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  7. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  8. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1
  9. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  10. 2026Adversarial Defense in Vision-Language Models: An Overviewcited by 1
  11. 2026ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Modelscited by 1
  12. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
10 more
  1. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  2. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  3. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  4. 2025FeatureFool: Zero-Query Fooling of Video Models via Feature Mapcited by 0
  5. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  6. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
  7. 2026AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Modelscited by 0
  8. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  9. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 0
  10. 2026Cross-device Collaborative Test-time Adaptation with Zeroth-order Optimization and Model Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  2. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  3. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  4. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
  5. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  6. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  7. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  8. 2025Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference