mlsanityData to 14 Oct 2026

Coordinated Robustness Evaluation Framework for Vision-Language Models

arXiv 2506.05429 ↗First posted 5 Jun 2025cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models, which integrate computer vision and natural language processing capabilities, have demonstrated significant advancements in tasks such as image captioning and visual question and answering. However, similar to traditional models, they are susceptible to small perturbations, posing a challenge to their robustness, particularly in deployment scenarios. Evaluating the robustness of these models requires perturbations in both the vision and language modalities to learn their inter-modal dependencies. In this work, we train a generic surrogate model that can take both image and text as input and generate joint representation which is further used to generate adversarial perturbations for both the text and image modalities. This coordinated attack strategy is evaluated on the visual question and answering and visual reasoning datasets using various state-of-the-art vision-language models. Our results indicate that the proposed strategy outperforms other multi-modal attacks and single-modality attacks from the recent literature. Our results demonstrate their effectiveness in compromising the robustness of several state-of-the-art pre-trained multi-modal models such as instruct-BLIP, ViLT and others.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 50 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

37 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective30 Apr
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  3. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 34

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
  11. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  12. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
22 more
  1. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  3. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  4. 2019Adversarial Attacks and Defenses in Images, Graphs and Text: A Reviewcited by 237
  5. 2020A Self-supervised Approach for Adversarial Robustnesscited by 180
  6. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  7. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  8. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  9. 2016Crafting Adversarial Input Sequences for Recurrent Neural Networkscited by 103
  10. 2022Natural Attack for Pre-trained Models of Codecited by 75
  11. 2020Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTcited by 69
  12. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 68
  13. 2019FDA: Feature Disruptive Attackcited by 56
  14. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 53
  15. 2021Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairscited by 45
  16. 2020CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generationcited by 44
  17. 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
  18. 2020Adversarial Attack and Defense of Structured Prediction Modelscited by 20
  19. 2023Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustnesscited by 7
  20. 2021R&R: Metric-guided Adversarial Sentence Generationcited by 6
  21. 2024Robustness and Visual Explanation for Black Box Image, Video, and ECG Signal Classification with Reinforcement Learningcited by 4
  22. 2025Reinforcement Learning Platform for Adversarial Black-box Attacks with Custom Distortion Filterscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  2. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
  3. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  4. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
  5. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  6. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  7. 2026Breaking the weakest link to evade vision language models
  8. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks