mlsanityData to 7 Oct 2026

Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models

arXiv 2508.05237 ↗First posted 7 Aug 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

This report synthesizes eight seminal papers on the zero-shot adversarial robustness of vision-language models (VLMs) like CLIP. A central challenge in this domain is the inherent trade-off between enhancing adversarial robustness and preserving the model's zero-shot generalization capabilities. We analyze two primary defense paradigms: Adversarial Fine-Tuning (AFT), which modifies model parameters, and Training-Free/Test-Time Defenses, which preserve them. We trace the evolution from alignment-preserving methods (TeCoA) to embedding space re-engineering (LAAT, TIMA), and from input heuristics (AOM, TTC) to latent-space purification (CLIPure). Finally, we identify key challenges and future directions including hybrid defense strategies and adversarial pre-training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 2 of its 9 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

2 of 9 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan

Other references · 1

  1. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  2. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models
  3. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  4. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  5. 2026Proxy Robustness in Vision Language Models is Effortlessly Transferable
  6. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  7. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
  8. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models