mlsanityData to 7 Oct 2026

Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models

arXiv 2410.21802 ↗First posted 29 Oct 2024cs.CV cs.AIcited by 23

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Due to the impressive zero-shot capabilities, pre-trained vision-language models (e.g. CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: the Attention Refinement module and the Attention-based Model Constraint module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness: The Attention Refinement module aligns the text-guided attention obtained from the target model via adversarial examples with the text-guided attention acquired from the original model via clean examples. This alignment enhances the model's robustness. Additionally, the Attention-based Model Constraint module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. The experiments validate that our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets. Our code is available at https://github.com/zhyblue424/TGA-ZSR.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 68 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

50 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  2. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  3. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  11. 2013Intriguing properties of neural networkscited by 5,114
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
35 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  13. 2019Adversarial Training for Free!cited by 638
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  16. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  17. 2022Diffusion Models for Adversarial Purificationcited by 469
  18. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  19. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  20. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  21. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  22. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  23. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  24. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  25. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  26. 2023Robust Evaluation of Diffusion-Based Adversarial Purificationcited by 65
  27. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  28. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  29. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  30. 2022Exploring Visual Interpretability for Contrastive Language-Image Pre-trainingcited by 29
  31. 2023Are Data-driven Explanations Robust against Out-of-distribution Data?cited by 14
  32. 2022Adversarial Purification with the Manifold Hypothesiscited by 8
  33. 2023Quality-agnostic Image Captioning to Safely Assist People with Vision Impairmentcited by 8
  34. 2023AROID: Improving Adversarial Robustness Through Online Instance-Wise Data Augmentationcited by 6
  35. 2024Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  2. 2025Seeing What Matters: Empowering CLIP with Patch Generation-to-Selectioncited by 7
  3. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  4. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  5. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  6. 2025Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Modelscited by 2
  7. 2026Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classificationcited by 2
  8. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  9. 2026Adversarial Defense in Vision-Language Models: An Overviewcited by 1
  10. 2026Continual Learning with Vision-Language Models via Semantic-Geometry Preservationcited by 1
  11. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
  12. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
11 more
  1. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  2. 2025LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoderscited by 0
  3. 2025Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weightingcited by 0
  4. 2026Observing Health Outcomes Using Remote Sensing Imagery and Geo-Context Guided Visual Transformercited by 0
  5. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  6. 2026Complementary Text-Guided Attention for Zero-Shot Adversarial Robustnesscited by 0
  7. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attackscited by 0
  8. 2026Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learningcited by 0
  9. 2026Unifying Adversarially Robust Model Experts in Vision-Language Modelscited by 0
  10. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0
  11. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  2. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  3. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models
  4. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  5. 2026Hierarchically Robust Zero-shot Vision-language Models
  6. 2024Toward a Holistic Evaluation of Robustness in CLIP Models
  7. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
  8. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models