mlsanityData to 7 Oct 2026

Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness

arXiv 2401.04350 ↗First posted 9 Jan 2024cs.CVcited by 48

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Large-scale pre-trained vision-language models like CLIP have demonstrated impressive performance across various tasks, and exhibit remarkable zero-shot generalization capability, while they are also vulnerable to imperceptible adversarial examples. Existing works typically employ adversarial training (fine-tuning) as a defense method against adversarial examples. However, direct application to the CLIP model may result in overfitting, compromising the model's capacity for generalization. In this paper, we propose Pre-trained Model Guided Adversarial Fine-Tuning (PMG-AFT) method, which leverages supervision from the original pre-trained model by carefully designing an auxiliary branch, to enhance the model's zero-shot adversarial robustness. Specifically, PMG-AFT minimizes the distance between the features of adversarial examples in the target model and those in the pre-trained model, aiming to preserve the generalization features already captured by the pre-trained model. Extensive Experiments on 15 zero-shot datasets demonstrate that PMG-AFT significantly outperforms the state-of-the-art method, improving the top-1 robust accuracy by an average of 4.99%. Furthermore, our approach consistently improves clean accuracy by an average of 8.72%. Our code is available at https://github.com/serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

35 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  3. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  4. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
19 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2021Robust fine-tuning of zero-shot modelscited by 785
  9. 2019Adversarial Training for Free!cited by 638
  10. 2020Overfitting in adversarially robust deep learningcited by 504
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  12. 2023A Cookbook of Self-Supervised Learningcited by 248
  13. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  14. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  15. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  16. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  17. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  18. 2023Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspectivecited by 18
  19. 2023Adversarial Attacks on Foundational Vision Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 19; 2026: 21.

48 papers.

Sort:

By area:

  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
  2. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Modelscited by 23
  3. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  4. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  5. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  6. 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 14
  7. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classificationcited by 12
  8. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Modelscited by 11
  9. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  10. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  11. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  12. 2024Adversarial Prompt Distillation for Vision-Language Modelscited by 6
36 more
  1. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  2. 2025Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Modelscited by 5
  3. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  4. 2024TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Abilitycited by 2
  5. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  6. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 2
  7. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 2
  8. 2025Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Modelscited by 2
  9. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  10. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationshipscited by 1
  11. 2025ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIPcited by 1
  12. 2025Calibrating Uncertainty for Zero-Shot Adversarial CLIPcited by 1
  13. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Modelscited by 1
  14. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
  15. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  16. 2026Hierarchically Robust Zero-shot Vision-language Modelscited by 1
  17. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  18. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Modelscited by 0
  19. 2025LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoderscited by 0
  20. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Modelscited by 0
  21. 2025Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weightingcited by 0
  22. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  23. 2026Complementary Text-Guided Attention for Zero-Shot Adversarial Robustnesscited by 0
  24. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Modelscited by 0
  25. 2026AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Modelscited by 0
  26. 2026TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Modelscited by 0
  27. 2026Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Modelscited by 0
  28. 2026T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Modelscited by 0
  29. 2026Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learningcited by 0
  30. 2026GeoDetect: Geometric Adversarial Detection for VLPscited by 0
  31. 2026Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Modelcited by 0
  32. 2026Unifying Adversarially Robust Model Experts in Vision-Language Modelscited by 0
  33. 2026ReACT-CLIP: Response-Aware Test-Time Defense for Vision--Language Modelscited by 0
  34. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  35. 2026Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Modelscited by 0
  36. 2026Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Robust Generalization of Adversarial Prompt Tuning
  2. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models
  3. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  4. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  5. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  6. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
  7. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  8. 2023Adversarial Prompt Tuning for Vision-Language Models