mlsanityData to 14 Oct 2026

Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models

arXiv 2601.10313 ↗First posted 15 Jan 2026cs.CV cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Existing adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a multimodal universal attack framework for VLP models. For the image modality, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, it hierarchically models textual importance by considering both intra- and inter-sentence contributions to identify globally influential words, which are then used as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and datasets, demonstrate the superior transferability of the proposed universal multimodal attacks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 54 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

43 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models12 Feb
  2. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models8 Jun
  3. 2024Universal Adversarial Perturbations for Vision-Language Pre-trained Models9 May
  4. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  5. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2016Adversarial examples in the physical worldcited by 1,769
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
26 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  5. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  6. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 536
  7. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  8. 2019Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attackscited by 442
  9. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 355
  10. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  11. 2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
  12. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 244
  13. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  14. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  15. 2021Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferabilitycited by 84
  16. 2023Structure Invariant Transformation for better Adversarial Transferabilitycited by 62
  17. 2021Boosting Adversarial Transferability through Enhanced Momentumcited by 54
  18. 2020Transferable Perturbations of Deep Feature Distributionscited by 51
  19. 2023An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferabilitycited by 42
  20. 2023Zero-Shot Learning by Harnessing Adversarial Samplescited by 24
  21. 2019Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defensescited by 23
  22. 2023Enhancing Generalization of Universal Adversarial Perturbation through Gradient Aggregationcited by 22
  23. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  24. 2021Universal Adversarial Training with Class-Wise Perturbationscited by 13
  25. 2024VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Modelscited by 10
  26. 2024Effective and Robust Adversarial Training against Data and Label Corruptionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach
  2. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Models
  3. 2026Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
  4. 2026VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models
  5. 2024Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
  6. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  7. 2024Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
  8. 2024A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models