mlsanityData to 14 Oct 2026

Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

arXiv 2408.03554 ↗First posted 7 Aug 2024cs.CL cs.CR cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 19 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

10 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  2. 2023Defense-Prefix for Preventing Typographic Attacks on CLIP10 Apr

Other references · 8

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  8. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 9.

17 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  3. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injectioncited by 44
  4. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  5. 2025Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Modelscited by 14
  6. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 13
  7. 2026Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructionscited by 8
  8. 2026VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Modelscited by 7
  9. 2024Goal Hijacking Attack on Large Language Models via Pseudo-Conversation Injectioncited by 4
  10. 2025AgentTypo: Adaptive Typographic Prompt Injection Attacks against Black-box Multimodal Agentscited by 4
  11. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Surveycited by 4
  12. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
5 more
  1. 2026Covert Visual Prompt Injection against Commercial Multimodal Large Language Modelscited by 0
  2. 2026A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoningcited by 0
  3. 2026Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agentscited by 0
  4. 2026VLAGuard: A Framework for Evaluating and Mitigating Physical Attention Hijacking in Vision-Language-Action Robots within Wireless Sensor Networkscited by 0
  5. 2026QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Physical Prompt Injection Attacks on Large Vision-Language Models
  2. 2025When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs
  3. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  4. 2025Text Prompt Injection of Vision Language Models
  5. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulated
  6. 2023InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
  7. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtime
  8. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift