mlsanityData to 7 Oct 2026

Attacks on multimodal models

arXiv 2412.01725 ↗First posted 2 Dec 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Today, models capable of working with various modalities simultaneously in a chat format are gaining increasing popularity. Despite this, there is an issue of potential attacks on these models, especially considering that many of them include open-source components. It is important to study whether the vulnerabilities of these components are inherited and how dangerous this can be when using such models in the industry. This work is dedicated to researching various types of attacks on such models and evaluating their generalization capabilities. Modern VLM models (LLaVA, BLIP, etc.) often use pre-trained parts from other models, so the main part of this research focuses on them, specifically on the CLIP architecture and its image encoder (CLIP-ViT) and various patch attack variations for it.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

29 of 42 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2016Universal adversarial perturbationscited by 774
  11. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
17 more
  1. 2017Adversarial Patchcited by 514
  2. 2024Large Language Models: A Surveycited by 506
  3. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  4. 2024An Introduction to Vision-Language Modelingcited by 175
  5. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 165
  6. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  7. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  8. 2019Unrestricted Adversarial Examples via Semantic Manipulationcited by 84
  9. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  10. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
  11. 2022Disentangling visual and written concepts in CLIPcited by 57
  12. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretrainingcited by 42
  13. 2022Natural Color Fool: Towards Boosting Black-box Unrestricted Attackscited by 38
  14. 2024Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attackscited by 36
  15. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Modelscited by 31
  16. 2022A Survey on Physical Adversarial Attack in Computer Visioncited by 24
  17. 2023Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognitioncited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models
  2. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey
  3. 2024Safety of Multimodal Large Language Models on Images and Texts
  4. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  5. 2026Jailbreaking Multimodal Large Language Models using Multi-Clip Video
  6. 2026VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts
  7. 2025Survey of Adversarial Robustness in Multimodal Large Language Models
  8. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models