mlsanityData to 7 Oct 2026

Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models

arXiv 2605.25922 ↗First posted 25 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision Language Models adapt well to downstream tasks but are highly vulnerable to adversarial perturbations that disrupt cross-modal semantic alignment. Existing defenses are largely unidirectional or structural, failing to exploit bidirectional cross-modal complementarity and instance-wise adaptive protection. To overcome the limitations of unidirectional and static defenses in adversarial settings, we propose Closed-Loop Bidirectional Prompting, casting robust adaptation as cross-modal agreement recovery via a dynamic feedback loop on frozen encoders. A Semantic Anchor is introduced as a stable prior to constrain cyclic updates and mitigate perturbation-induced feature corruption. Through anchor-based bootstrapping, textual semantics denoise visual representations, while the refined visuals enable instance-adaptive prompt updating, yielding a rectified and robust consensus. Extensive evaluations across 11 datasets validate state-of-the-art robustness and strong base-to-new generalization, while maintaining a favorable trade-off between computational cost and accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 49 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

38 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models26 Oct
  2. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  3. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models3 Feb
  4. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  5. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar
  6. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  7. 2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness9 Jan
  8. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  9. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
17 more
  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  8. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  9. 2021Extract Free Dense Labels from CLIPcited by 580
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  11. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  12. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  13. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 65
  14. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51
  15. 2021Combating Adversaries with Anti-Adversariescited by 27
  16. 2025CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIPcited by 23
  17. 2025KPL: Training-Free Medical Knowledge Mining of Vision-Language Modelscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  2. 2026Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
  3. 2026SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
  4. 2025NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models
  5. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models
  6. 2025Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise
  7. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  8. 2024Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens