mlsanityData to 14 Oct 2026

TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models

arXiv 2605.17577 ↗First posted 17 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

Large-scale pre-trained Vision-Language models (VLMs), such as CLIP, exhibit strong zero-shot generalization, yet remain highly vulnerable to imperceptible adversarial perturbations, raising serious safety concerns for open-world deployment. To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense. Building upon our prior Test-Time Adversarial Prompt Tuning (TAPT), TAME introduces an architectural reformulation by replacing TAPT's single adaptive prompt with an input-conditioned Mixture-of-Experts (MoE) framework, enabling more expressive and adaptive defense. Specifically, TAME maintains a bank of learnable expert prompts and employs an input-dependent routing mechanism to aggregate a customized prompt mixture for each unlabeled test sample at inference time. This test-time defense mechanism is driven by three unsupervised objectives: (1) multi-view prediction entropy minimization, (2) layer-wise alignment of visual token statistics to precomputed clean and adversarial reference distributions, and (3) MoE regularization for balanced expert utilization and prompt diversity. We evaluated TAME on 11 benchmark datasets, including ImageNet and 10 additional zero-shot datasets. The results show that TAME improves the zero-shot adversarial robustness of the original CLIP by at least 49.1% under AutoAttack while largely preserving generalization on clean samples. TAME also consistently outperforms existing adversarial prompt tuning methods across multiple prompt designs, yielding an average robustness gain of at least 30.2%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 83 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

64 of 83 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs4 Jan
  2. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models15 Jun
  3. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety2 Feb
  5. 2024Adversarial Prompt Distillation for Vision-Language Models22 Nov
  6. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models20 Nov
  7. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  8. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  9. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  10. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  11. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  12. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
8 more
  1. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  2. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  3. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  4. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning24 Aug
  5. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  6. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  7. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  8. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2013Describing Textures in the Wildcited by 2,125
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
32 more
  1. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  5. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  7. 2022Continual Test-Time Domain Adaptationcited by 525
  8. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  9. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  11. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377
  12. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  13. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  14. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 236
  15. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  16. 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
  17. 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
  18. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  19. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 101
  20. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  21. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  22. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 59
  23. 2024Revisiting Adversarial Training at Scalecited by 33
  24. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  25. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 21
  26. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learningcited by 17
  27. 2024One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Promptscited by 16
  28. 2023Multi-event Video-Text Retrievalcited by 15
  29. 2020Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustnesscited by 11
  30. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 8
  31. 2024AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuningcited by 7
  32. 2025Imperceptible Jailbreaking against Large Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  2. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
  3. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  4. 2025MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
  5. 2025MINT: Memory-Infused Prompt Tuning at Test-time for CLIP
  6. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  7. 2026Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
  8. 2025Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models