mlsanityData to 7 Oct 2026

Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques

arXiv 2407.11121 ↗First posted 15 Jul 2024cs.CV cs.AI cs.LGcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-Language Models (VLMs) have witnessed a surge in both research and real-world applications. However, as they are becoming increasingly prevalent, ensuring their robustness against adversarial attacks is paramount. This work systematically investigates the impact of model design choices on the adversarial robustness of VLMs against image-based attacks. Additionally, we introduce novel, cost-effective approaches to enhance robustness through prompt formatting. By rephrasing questions and suggesting potential adversarial perturbations, we demonstrate substantial improvements in model robustness against strong image-based attacks such as Auto-PGD. Our findings provide important guidelines for developing more robust VLMs, particularly for deployment in safety-critical environments.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 32 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

28 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  2. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attacks6 Dec
  3. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  4. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
12 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  4. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  5. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  6. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  7. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  8. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  9. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  10. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  11. 2024A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)cited by 44
  12. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answeringcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Modelscited by 9
  2. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  3. 2026PA-Attack: Guiding Gray-Box Attacks on LVLM Vision Encoders with Prototypes and Attentioncited by 3
  4. 2026CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Modelscited by 0
  5. 2026Investigating Adversarial Robustness of Multi-modal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  2. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  3. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  4. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  5. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  6. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
  7. 2024Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey
  8. 2026Breaking the weakest link to evade vision language models