mlsanityData to 14 Oct 2026

AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models

arXiv 2605.15584 ↗First posted 15 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models like CLIP have demonstrated remarkable zero-shot transfer capabilities. However, their susceptibility to imperceptible adversarial perturbations remains a critical security concern. While test-time defenses offer a pragmatic solution for deployed models, existing approaches typically rely on gradient-based optimization during inference, incurring significant computational overhead. In this paper, we revisit the role of data augmentation in CLIP robustness and observe that augmentations are not equally effective: specific augmentations consistently provide robust geometric cues that align with correct class semantics in the hyperspherical feature space. Based on this, we propose Adaptive Geodesic Correction (AGC), a training-free defense mechanism that requires no parameter updates. AGC identifies a reliable augmentation as a geometric anchor and corrects the input feature towards it, utilizing an adaptive step size to balance robustness against clean accuracy preservation. AGC achieves superior performance across eight fine-grained datasets and three CLIP backbones, improving average robust accuracy by 44.4% over state-of-the-art baseline while delivering a 10× reduction in inference latency. Our findings reveal a fundamental geometric property of CLIP features, offering a highly efficient and effective paradigm for robust multimodal deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 40 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

32 of 40 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models18 Dec
  2. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  3. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models20 Nov
  4. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  5. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  6. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  7. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  8. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2016Adversarial examples in the physical worldcited by 1,769
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  12. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
12 more
  1. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  3. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  4. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 154
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2021Feature Importance-aware Transferable Adversarial Attackscited by 127
  7. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 101
  8. 2022Improving Adversarial Transferability via Neuron Attribution-Based Attackscited by 93
  9. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  10. 2023Boosting Adversarial Transferability by Block Shuffle and Rotationcited by 64
  11. 2024Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Modelscited by 32
  12. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  2. 2026Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
  3. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models
  4. 2026Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
  5. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  6. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  7. 2026Proxy Robustness in Vision Language Models is Effortlessly Transferable
  8. 2024Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation