mlsanityData to 14 Oct 2026

R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning

arXiv 2504.11195 ↗First posted 15 Apr 2025cs.LG cs.CR cs.CVcited by 33

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs), such as CLIP, have gained significant popularity as foundation models, with numerous fine-tuning methods developed to enhance performance on downstream tasks. However, due to their inherent vulnerability and the common practice of selecting from a limited set of open-source models, VLMs suffer from a higher risk of adversarial attacks than traditional vision models. Existing defense techniques typically rely on adversarial fine-tuning during training, which requires labeled data and lacks of flexibility for downstream tasks. To address these limitations, we propose robust test-time prompt tuning (R-TPT), which mitigates the impact of adversarial attacks during the inference stage. We first reformulate the classic marginal entropy objective by eliminating the term that introduces conflicts under adversarial conditions, retaining only the pointwise entropy minimization. Furthermore, we introduce a plug-and-play reliability-based weighted ensembling strategy, which aggregates useful information from reliable augmented views to strengthen the defense. R-TPT enhances defense against adversarial attacks without requiring labeled training data while offering high flexibility for inference tasks. Extensive experiments on widely used benchmarks with various attacks demonstrate the effectiveness of R-TPT. The code is available in https://github.com/TomSheng21/R-TPT.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 62 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

50 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  3. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  11. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
35 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  6. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  8. 2016Universal adversarial perturbationscited by 784
  9. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  10. 2021Extract Free Dense Labels from CLIPcited by 583
  11. 2017One pixel attack for fooling deep neural networkscited by 582
  12. 2022Continual Test-Time Domain Adaptationcited by 525
  13. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  14. 2022Diffusion Models for Adversarial Purificationcited by 476
  15. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  16. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377
  17. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  18. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 359
  19. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  20. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  21. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 154
  22. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  23. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  24. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 101
  25. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  26. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 83
  27. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  28. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  29. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  30. 2023Image Clustering with External Guidancecited by 36
  31. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 32
  32. 2022Semantic-Enhanced Image Clusteringcited by 25
  33. 2023Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classificationcited by 17
  34. 2024STAMP: Outlier-Aware Test-Time Adaptation with Stable Memory Replaycited by 13
  35. 2023AdaptGuard: Defending Against Universal Attacks for Model Adaptationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 23.

33 papers.

Sort:

By area:

  1. 2025Towards Generalizable AI-Generated Image Detection via Image-Adaptive Prompt Learningcited by 9
  2. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 7
  3. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  4. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 3
  5. 2025D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Modelscited by 3
  6. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  7. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 3
  8. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  9. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  10. 2026Fed-ADE: Adaptive Learning Rate for Federated Post-adaptation under Distribution Shiftcited by 2
  11. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  12. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1
21 more
  1. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  2. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  3. 2026Adversarial Defense in Vision-Language Models: An Overviewcited by 1
  4. 2026Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Modelscited by 1
  5. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Modelscited by 1
  6. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
  7. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  8. 2026FACTOR: Counterfactual Training-Free Test-Time Adaptation for Open-Vocabulary Object Detectioncited by 1
  9. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  10. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  11. 2026XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMscited by 0
  12. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
  13. 2026AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Modelscited by 0
  14. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  15. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Modelscited by 0
  16. 2026Investigating Adversarial Robustness of Multi-modal Large Language Modelscited by 0
  17. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Modelscited by 0
  18. 2026T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Modelscited by 0
  19. 2026Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Modelscited by 0
  20. 2026Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuningcited by 0
  21. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  2. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  3. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  4. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models
  5. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  6. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  7. 2023Adversarial Prompt Tuning for Vision-Language Models
  8. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models