mlsanityData to 7 Oct 2026

PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks

arXiv 2604.01010 ↗First posted 1 Apr 2026cs.CV cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) are vulnerable to adversarial image perturbations. Existing works based on adversarial training against task-specific adversarial examples are computationally expensive and often fail to generalize to unseen attack types. To address these limitations, we introduce Paraphrase-Decomposition-Aggregation (PDA), a training-free defense framework that leverages text augmentation to enhance VLM robustness under diverse adversarial image attacks. PDA performs prompt paraphrasing, question decomposition, and consistency aggregation entirely at test time, thus requiring no modification on the underlying models. To balance robustness and efficiency, we instantiate PDA as invariants that reduce the inference cost while retaining most of its robustness gains. Experiments on multiple VLM architectures and benchmarks for visual question answering, classification, and captioning show that PDA achieves consistent robustness gains against various adversarial perturbations while maintaining competitive clean accuracy, establishing a generic, strong and practical defense framework for VLMs during inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 67 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

52 of 67 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning15 Apr
  2. 2024Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks24 Nov
  3. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models20 Nov
  4. 2024Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models29 Oct
  5. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models7 Oct
  6. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Models21 Mar
  7. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  8. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  9. 2024Safety of Multimodal Large Language Models on Images and Texts1 Feb
  10. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  11. 2023Adversarial Prompt Tuning for Vision-Language Models19 Nov
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
1 more
  1. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 39

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
27 more
  1. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  2. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  3. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  4. 2018Robustness May Be at Odds with Accuracycited by 873
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  9. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  10. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
  11. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  12. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  13. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  14. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  15. 2020Randomized Smoothing of All Shapes and Sizescited by 146
  16. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  17. 2020MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radiuscited by 114
  18. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  19. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  20. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  21. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  22. 2024One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Promptscited by 16
  23. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learningcited by 16
  24. 2023Incremental Randomized Smoothing Certificationcited by 12
  25. 2022Localized Randomized Smoothing for Collective Robustness Certificationcited by 10
  26. 2022(De-)Randomized Smoothing for Decision Stump Ensemblescited by 5
  27. 2023Fast Certification of Vision-Language Models Using Incremental Randomized Smoothingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Coordinated Robustness Evaluation Framework for Vision-Language Models
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  4. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  5. 2025Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
  6. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  7. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  8. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques