mlsanityData to 7 Oct 2026

Understanding Zero-Shot Adversarial Robustness for Large-Scale Models

arXiv 2212.07016 ↗First posted 14 Dec 2022cs.CVcited by 117

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks. Yet imperceptible adversarial perturbations can significantly reduce CLIP's performance on new tasks. In this work, we identify and explore the problem of adapting large-scale models for zero-shot adversarial robustness. We first identify two key factors during model adaption -- training losses and adaptation methods -- that affect the model's zero-shot adversarial robustness. We then propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning on a small set of training data. We apply this training loss to two adaption methods, model finetuning and visual prompt tuning. We find that visual prompt tuning is more effective in the absence of texts, while finetuning wins in the existence of text guidance. Overall, our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement of over 31 points over ImageNet and 15 zero-shot datasets. We hope this work can shed light on understanding the zero-shot adversarial robustness of large-scale models.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 76 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

53 of 76 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2022Visual Prompt Tuningcited by 1,674
38 more
  1. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  7. 2018Robustness May Be at Odds with Accuracycited by 873
  8. 2021Robust fine-tuning of zero-shot modelscited by 785
  9. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  10. 2020Overfitting in adversarially robust deep learningcited by 504
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  13. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  14. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  15. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  16. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  17. 2022Visual Prompting via Image Inpaintingcited by 248
  18. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  19. 2020Bag of Tricks for Adversarial Trainingcited by 199
  20. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  21. 2018Disentangling Adversarial Robustness and Generalizationcited by 171
  22. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  23. 2019Are Labels Required for Improving Adversarial Robustness?cited by 162
  24. 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
  25. 2020Adversarial Self-Supervised Contrastive Learningcited by 138
  26. 2018Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Modelscited by 135
  27. 2019Metric Learning for Adversarial Robustnesscited by 77
  28. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
  29. 2018Generative Dual Adversarial Network for Generalized Zero-shot Learningcited by 61
  30. 2021Adversarial Attacks are Reversible with Natural Supervisioncited by 47
  31. 2019What it Thinks is Important is Important: Robustness Transfers through Input Gradientscited by 34
  32. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34
  33. 2019Dual Adversarial Semantics-Consistent Network for Generalized Zero-Shot Learningcited by 20
  34. 2022Landscape Learning for Neural Network Inversioncited by 15
  35. 2019Bidirectional Inference Networks: A Class of Deep Bayesian Networks for Health Profilingcited by 11
  36. 2020Zero-Shot Learning from Adversarial Feature Residual to Compact Visual Featurecited by 10
  37. 2022Robust Perception through Equivariancecited by 9
  38. 2019A Meta-Learning Framework for Generalized Zero-Shot Learningcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 22; 2025: 38; 2026: 41.

117 papers.

Sort:

By area:

  1. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 119
  2. 2023A Pilot Study of Query-Free Adversarial Attack against Stable Diffusioncited by 75
  3. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
  4. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  5. 2023Adversarial Prompt Tuning for Vision-Language Modelscited by 42
  6. 2023CLIP in Medical Imaging: A Surveycited by 40
  7. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  8. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  9. 2023Visual Tuningcited by 34
  10. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  11. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
  12. 2023Language-Driven Anchors for Zero-Shot Adversarial Robustnesscited by 23
38 more
  1. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  2. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  3. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  4. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  5. 2025X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIPcited by 21
  6. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19
  7. 2023Convolutional Visual Prompt for Robust Visual Perceptioncited by 17
  8. 2023Foundation Model is Efficient Multimodal Multitask Model Selectorcited by 16
  9. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  10. 2023Adversarial Attacks on Foundational Vision Modelscited by 13
  11. 2024Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Modelscited by 12
  12. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  13. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 12
  14. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Modelscited by 12
  15. 2023Interpreting and Controlling Vision Foundation Models via Text Explanationscited by 11
  16. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Modelscited by 11
  17. 2023CLAP: Isolating Content from Style through Contrastive Learning with Augmented Promptscited by 10
  18. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustnesscited by 10
  19. 2025Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metricscited by 10
  20. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Modelscited by 10
  21. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  22. 2023Multi-scale Diffusion Denoised Smoothingcited by 9
  23. 2024Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Modelscited by 9
  24. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Modelscited by 9
  25. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
  26. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  27. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  28. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  29. 2023Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Modelscited by 6
  30. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
  31. 2025OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triadcited by 6
  32. 2024Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformerscited by 5
  33. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniquescited by 5
  34. 2024AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?cited by 5
  35. 2024Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Modelscited by 5
  36. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  37. 2026Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learningcited by 5
  38. 2024Unaligning Everything: Or Aligning Any Text to Any Image in Multimodal Modelscited by 3

50 most cited of 117 All 117 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  2. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  3. 2025Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models
  4. 2024Revisiting the Robust Generalization of Adversarial Prompt Tuning
  5. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  6. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models
  7. 2026Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
  8. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models