mlsanityData to 14 Oct 2026

T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models

arXiv 2606.23132 ↗First posted 22 Jun 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) achieve strong zero-shot recognition, but they remain highly vulnerable to adversarial perturbations. Recent test-time adaptations improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself. Prompt-based methods adapt the learnable text prompts, while input-space methods optimize pixels or padding at test time. These approaches can improve predictions, but they do so through an indirect and expensive optimization path. We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space. T-VSS first builds a sample-specific low-rank subspace from multi-view feature residuals anchored at the attacked image. It then learns a shared feature correction within this subspace using reliability-weighted entropy minimization. By constraining adaptation to a compact visual geometry, T-VSS steers attacked features toward more stable and discriminative predictions while avoiding noisy full-space updates. Experiments on fine-grained, ImageNet, and ImageNet-OOD benchmarks show that T-VSS improves adversarial robustness while maintaining competitive clean accuracy and better efficiency than prior test-time adaptations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 57 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

46 of 57 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models4 Jun
  2. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models31 Mar
  3. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models18 Dec
  4. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  5. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  6. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  7. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  8. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  9. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  10. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2016Adversarial examples in the physical worldcited by 1,769
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
24 more
  1. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  6. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  9. 2016Universal adversarial perturbationscited by 784
  10. 2020Overfitting in adversarially robust deep learningcited by 506
  11. 2022Diffusion Models for Adversarial Purificationcited by 476
  12. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  13. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
  14. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 236
  15. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  16. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 83
  17. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  18. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  19. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  20. 2024CLIP-Guided Generative Networks for Transferable Targeted Adversarial Attackscited by 19
  21. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
  22. 2025Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Modelscited by 3
  23. 2026StreamTTO: Efficient Online Test-Time Optimization for Video Depth Completioncited by 3
  24. 2026Semantic-aware Adversarial Fine-tuning for CLIPcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
  2. 2026Subspace Alignment for Vision-Language Model Test-time Adaptation
  3. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  4. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Models
  5. 2026Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
  6. 2026Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
  7. 2024Towards Evaluating the Robustness of Visual State Space Models
  8. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models