mlsanityData to 14 Oct 2026

Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

arXiv 2510.02913 ↗First posted 3 Oct 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models such as CLIP demonstrate impressive zero-shot generalization but remain highly vulnerable to adversarial attacks. Prior adversarial methods treat all samples equally in the loss function, despite the fact that not all inputs contribute equally to adversarial vulnerability. Some samples retain high confidence even under large perturbations, while others are unstable and change predictions with minor noise. To address this, we propose Confidence-Aware Weighting (CAW) to enhance the zero-shot robustness of vision-language models. CAW introduces two key components: (1) a Confidence-Aware Loss that prioritizes uncertain adversarial examples by scaling the KL divergence between clean and adversarial predictions, and (2) a Feature Alignment Regularization that preserves semantic consistency by minimizing the distance between frozen and fine-tuned image encoder features on adversarial inputs. Together, these components improve both clean and robust accuracy without compromising generalization. Extensive experiments on TinyImageNet and 14 additional datasets show that CAW outperforms recent state-of-the-art methods such as PMG-AFT and TGA-ZSR under strong adversarial attacks like AutoAttack, while also being more memory-efficient.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 56 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

38 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models29 Oct
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  3. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  5. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  12. 2013Intriguing properties of neural networkscited by 5,155
21 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  5. 2015Fast R-CNNcited by 2,933
  6. 2016Adversarial examples in the physical worldcited by 1,769
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  11. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  12. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  13. 2019Adversarial Training for Free!cited by 643
  14. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  15. 2018Stochastic Activation Pruning for Robust Adversarial Defensecited by 171
  16. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  17. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  18. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  19. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
  20. 2022Improving Adversarial Robustness by Putting More Regularizations on Less Robust Samplescited by 11
  21. 2018Security Analysis and Enhancement of Model Compressed Deep Learning Systems under Adversarial Attackscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  2. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models
  3. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  4. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  5. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
  6. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  7. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  8. 2026Proxy Robustness in Vision Language Models is Effortlessly Transferable