mlsanityData to 14 Oct 2026

When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustness

arXiv 2606.06938 ↗First posted 5 Jun 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models such as CLIP have achieved remarkable zero-shot recognition capabilities, yet their robustness against adversarial perturbations remains limited. Test-time counterattack (TTC) was recently proposed to improve CLIP's robustness by perturbing an input image to steer it away from a corrupted state during inference. However, TTC remains fragile under strong attacks because its counterattack relies on a directly corrupted original view and employs a noise-driven hard-gating scheme that cannot adapt to varying corruption severity. To address these limitations, we introduce Multi-view guided Adaptive Counterattack (MAC), which performs counterattacks for multi-view with corruption-aware soft weighting. Specifically, MAC first constructs augmented views of an input image to obtain diverse embeddings. It then performs counterattacks to refine corrupted embeddings of views. Next, MAC adaptively scales the counterattack intensity for each view based on its estimated corruption degree. Finally, the adaptively counterattacked views are aggregated to yield a robust final prediction. Extensive experiments across 20 datasets and diverse attack scenarios demonstrate that MAC substantially improves robustness while preserving high inference speed and memory efficiency with its tuning-free design. Our code is available at https://github.com/sunoh-kim/MAC.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 96 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

74 of 96 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP27 Jan
  2. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  3. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
  4. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models20 Nov
  5. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  6. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  7. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  8. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  9. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  10. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  11. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  12. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
38 more
  1. 2016Adversarial examples in the physical worldcited by 1,769
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  7. 2019Natural Adversarial Examplescited by 1,408
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  9. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  13. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  14. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  15. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
  16. 2016Universal adversarial perturbationscited by 784
  17. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  18. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  19. 2019Adversarial Training for Free!cited by 643
  20. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  21. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  22. 2017One pixel attack for fooling deep neural networkscited by 582
  23. 2022Continual Test-Time Domain Adaptationcited by 525
  24. 2020Overfitting in adversarially robust deep learningcited by 506
  25. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  26. 2022Diffusion Models for Adversarial Purificationcited by 476
  27. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  28. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  29. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  30. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  31. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
  32. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
  33. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 141
  34. 2021Adversarial purification with Score-based generative modelscited by 134
  35. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  36. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  37. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  38. 2021Attacking Adversarial Attacks as A Defensecited by 32

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference
  2. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
  3. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  4. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  5. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
  6. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  7. 2024Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
  8. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models