mlsanityData to 7 Oct 2026

Extended to Reality: Prompt Injection in 3D Environments

arXiv 2602.07104 ↗First posted 6 Feb 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have advanced the capabilities to interpret and act on visual input in 3D environments, empowering diverse applications such as robotics and situated conversational agents. When MLLMs reason over camera-captured views of the physical world, a new attack surface emerges: an attacker can place text-bearing physical objects in the environment to override MLLMs' intended task. While prior work has studied prompt injection in the text domain and through digitally edited 2D images, limited attention has been paid to how these attacks function in 3D environments. To bridge the gap, we introduce PI3D, a prompt injection attack against MLLMs in 3D environments, realized through text-bearing object placement rather than digital image edits. We formulate and solve the problem of identifying an effective pose (position and orientation) for a 3D object with injected text, where the attacker's goal is to induce the MLLM to perform the injected task while ensuring that the object placement remains physically plausible. Experiment results demonstrate that PI3D is an effective attack against multiple MLLMs under diverse camera trajectories. We further evaluate a range of defenses and show that they are not sufficient to reliably defend against PI3D.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 65 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

39 of 65 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Physical Prompt Injection Attacks on Large Vision-Language Models24 Jan
  2. 2025Manipulating Multimodal Agents via Cross-Modal Prompt Injection19 Apr
  3. 2025ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality22 Jan
  4. 2024SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments28 Nov
  5. 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors17 May

Other references · 34

  1. 2016Adversarial examples in the physical worldcited by 1,762
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  5. 2017Adversarial Patchcited by 514
  6. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  7. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  8. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  9. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  10. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  11. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  12. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
22 more
  1. 2019Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Drivingcited by 151
  2. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  3. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  4. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  5. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  6. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  7. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  8. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 105
  9. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  10. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  11. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  12. 2025Agent Learning via Early Experiencecited by 79
  13. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  14. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  15. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  16. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  17. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  18. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  19. 2016Defining the Pose of any 3D Rigid Object and an Associated Distancecited by 17
  20. 2025Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Modelscited by 12
  21. 2023SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graphcited by 6
  22. 2025LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robots
  2. 2026Covert Visual Prompt Injection against Commercial Multimodal Large Language Models
  3. 2026Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models
  4. 2026Render Before Reading: Visual Rendering as a Prompt Injection Defense
  5. 2026On the Adversarial Robustness of 3D Large Vision-Language Models
  6. 2025When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs
  7. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation
  8. 2026Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions