mlsanityData to 7 Oct 2026

Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens

arXiv 2406.13294 ↗First posted 19 Jun 2024cs.MM cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models (VLMs) seamlessly integrate visual and textual data to perform tasks such as image classification, caption generation, and visual question answering. However, adversarial images often struggle to deceive all prompts effectively in the context of cross-prompt migration attacks, as the probability distribution of the tokens in these images tends to favor the semantics of the original image rather than the target tokens. To address this challenge, we propose a Contextual-Injection Attack (CIA) that employs gradient-based perturbation to inject target tokens into both visual and textual contexts, thereby improving the probability distribution of the target tokens. By shifting the contextual semantics towards the target tokens instead of the original image semantics, CIA enhances the cross-prompt transferability of adversarial images.Extensive experiments on the BLIP2, InstructBLIP, and LLaVA models show that CIA outperforms existing methods in cross-prompt transferability, demonstrating its potential for more effective adversarial strategies in VLMs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 35 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
11 more
  1. 2018Exploring Visual Relationship for Image Captioningcited by 215
  2. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  3. 2022SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustnesscited by 74
  4. 2019Enhancing Cross-task Black-Box Transferability of Adversarial Examples with Dispersion Reductioncited by 47
  5. 2017Visual Question Generation as Dual Task of Visual Question Answeringcited by 41
  6. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  7. 2022Character-level White-Box Adversarial Attacks against Transformers via Attachable Subwords Substitutioncited by 28
  8. 2021Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Frameworkcited by 27
  9. 2021Textual Backdoor Attacks Can Be More Harmful via Two Simple Trickscited by 17
  10. 2023CT-GAT: Cross-Task Generative Adversarial Attack based on Transferabilitycited by 6
  11. 2023Boosting Cross-task Transferability of Adversarial Patches with Visual Relationscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  3. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  4. 2025Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Models
  3. 2026Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction
  4. 2024Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks
  5. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
  6. 2026On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
  7. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
  8. 2026Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models