mlsanityData to 7 Oct 2026

Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP

arXiv 2601.19210 ↗First posted 27 Jan 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defenses are promising, existing methods fail to provide sufficient robustness against strong attacks and are often hampered by high inference latency and task-specific applicability. To address these limitations, we start by investigating the intrinsic properties of AEs, which reveals that AEs exhibit severe feature inconsistency under progressive frequency attenuation. We further attribute this to the model's inherent spectral bias. Leveraging this insight, we propose an efficient test-time defense named Contrastive Spectral Rectification (CSR). CSR optimizes a rectification perturbation to realign the input with the natural manifold under a spectral-guided contrastive objective, which is applied input-adaptively. Extensive experiments across 16 classification benchmarks demonstrate that CSR outperforms the SOTA by an average of 18.1% against strong APGD with modest inference overhead. Furthermore, CSR exhibits broad applicability across diverse visual tasks. Code is available at https://github.com/Summu77/CSR.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 99 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

67 of 99 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs25 Nov
  2. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  3. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o113 Mar
  4. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models20 Nov
  5. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  6. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks11 Sep
  7. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  8. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  9. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  10. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  11. 2024Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory19 Mar
  12. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
11 more
  1. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  2. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  3. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  4. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  5. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  6. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  7. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning24 Aug
  8. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  9. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  10. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  11. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  11. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
32 more
  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2021Intriguing Properties of Vision Transformerscited by 446
  4. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  5. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  6. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  7. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  8. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  9. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 101
  10. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 100
  11. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  12. 2025Test-Time Learning for Large Language Modelscited by 46
  13. 2025Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignmentcited by 42
  14. 2024Revisiting Adversarial Training at Scalecited by 33
  15. 2021Attacking Adversarial Attacks as A Defensecited by 31
  16. 2021Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neuronscited by 29
  17. 2021A Frequency Perspective of Adversarial Robustnesscited by 28
  18. 2021Combating Adversaries with Anti-Adversariescited by 27
  19. 2021Enhancing Adversarial Robustness via Test-time Transformation Ensemblingcited by 27
  20. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  21. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  22. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learningcited by 16
  23. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  24. 2023Towards Building More Robust Models with Frequency Biascited by 12
  25. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  26. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  27. 2025Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimizationcited by 11
  28. 2020Adversarial Image Color Transformations in Explicit Color Filter Spacecited by 9
  29. 2018A Spectral View of Adversarially Robust Featurescited by 4
  30. 2025D-TPT: Dimensional Entropy Maximization for Calibrating Test-Time Prompt Tuning in Vision-Language Modelscited by 2
  31. 2025FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Modelscited by 2
  32. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  2. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  2. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  3. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
  4. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  5. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  6. 2025Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference
  7. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models
  8. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models