mlsanityData to 7 Oct 2026

VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models

arXiv 2310.04655 ↗First posted 7 Oct 2023cs.CR cs.CVcited by 66

Not on the map. Why?

Cited by 48 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multimodal levels. At the single-modal level, we propose a new block-wise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multimodal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack five widely-used VL pre-trained models for six tasks. Experimental results show that VLATTACK achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a blind spot in the deployment of pre-trained VL models. Source codes can be found at https://github.com/ericyinyzy/VLAttack.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 53 of its 60 references that are in the map. First posted on arXiv 7 Oct 2023.

Builds on

53 of 60 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2016Adversarial Machine Learning at Scalecited by 934
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  9. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  12. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  13. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  14. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 440
  15. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  16. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  17. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  18. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  19. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  20. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  21. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  22. 2020A Self-supervised Approach for Adversarial Robustnesscited by 180
  23. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  24. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 165
  25. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  26. 2019Enhancing Adversarial Example Transferability with an Intermediate Level Attackcited by 139
  27. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  28. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  29. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  30. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  31. 2013Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networkscited by 84
  32. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  33. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  34. 2020Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferabilitycited by 57
  35. 2019FDA: Feature Disruptive Attackcited by 55
  36. 2020Transferable Perturbations of Deep Feature Distributionscited by 51
  37. 2019Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reductioncited by 47
  38. 2022Boosting the Transferability of Adversarial Attacks with Global Momentum Initializationcited by 29

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 22; 2026: 22.

66 papers.

Sort:

By area:

  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  2. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  3. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  4. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  5. 2024Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Modelscited by 48
  6. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 44
  7. 2024AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Modelscited by 43
  8. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentationcited by 39
  9. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
  10. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  11. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 35
  12. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
38 more
  1. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Modelscited by 21
  2. 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operativecited by 21
  3. 2024MirrorCheck: Efficient Adversarial Defense for Vision-Language Modelscited by 19
  4. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  5. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  6. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulatedcited by 12
  7. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Modelscited by 10
  8. 2025AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Deliverycited by 9
  9. 2025V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMscited by 9
  10. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approachcited by 7
  11. 2025QAVA: Query-Agnostic Visual Attack to Large Vision-Language Modelscited by 7
  12. 2025VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Modelscited by 7
  13. 2023Asymmetric Bias in Text-to-Image Generation with Adversarial Attackscited by 5
  14. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniquescited by 5
  15. 2025Universal Adversarial Attack on Aligned Multimodal LLMscited by 5
  16. 2024Adversarial Attacks on Both Face Recognition and Face Anti-spoofing Modelscited by 4
  17. 2024Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Modelscited by 4
  18. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  19. 2025Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Frameworkcited by 4
  20. 2025CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMscited by 3
  21. 2025GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbationscited by 3
  22. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 3
  23. 2025MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Modelscited by 2
  24. 2025Hydra: An Agentic Reasoning Approach for Enhancing Adversarial Robustness and Mitigating Hallucinations in Vision-Language Modelscited by 2
  25. 2025CoDefend: Cross-Modal Collaborative Defense via Diffusion Purification and Prompt Optimizationcited by 2
  26. 2025Sequential Comics for Jailbreaking Multimodal Large Language Models via Structured Visual Storytellingcited by 2
  27. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  28. 2025Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updatescited by 1
  29. 2025Adversarial Video Promotion Against Text-to-Video Retrievalcited by 1
  30. 2025High-Entropy Tokens as Multimodal Failure Points in Vision-Language Modelscited by 1
  31. 2025Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMscited by 0
  32. 2025Coordinated Robustness Evaluation Framework for Vision-Language Modelscited by 0
  33. 2025Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Modelscited by 0
  34. 2025Pay Less Attention to Function Words for Free Robustness of Vision-Language Modelscited by 0
  35. 2026On the Adversarial Robustness of 3D Large Vision-Language Modelscited by 0
  36. 2026Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluationcited by 0
  37. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0
  38. 2026Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patch for Infrared Vision-Language Modelscited by 0

50 most cited of 66 All 66 (JSON)