# If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems

arXiv: https://arxiv.org/abs/2604.19844 · HTML: https://mlsanity.com/paper/2604.19844/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2205.07460/) (2022)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [A Self-supervised Approach for Adversarial Robustness](https://mlsanity.com/paper/2006.04924/) (2020)
- [Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making](https://mlsanity.com/paper/2410.07166/) (2024)
- [DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2504.11358/) (2025)
- [Dissecting Adversarial Robustness of Multimodal LM Agents](https://mlsanity.com/paper/2406.12814/) (2024)
- [A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems](https://mlsanity.com/paper/2402.18649/) (2024)
- [Multimodal Situational Safety](https://mlsanity.com/paper/2410.06172/) (2024)
- [BadRobot: Jailbreaking Embodied LLM Agents in the Physical World](https://mlsanity.com/paper/2407.20242/) (2024)
- [SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments](https://mlsanity.com/paper/2412.00114/) (2024)
- [CHAI: Command Hijacking against embodied AI](https://mlsanity.com/paper/2510.00181/) (2025)
- [Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments](https://mlsanity.com/paper/2502.06445/) (2025)
- [Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones](https://mlsanity.com/paper/2311.15033/) (2023)
- [Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation](https://mlsanity.com/paper/2501.18416/) (2025)

## Built on by (most cited first, 1 of 1)

- [Security of Foundation-Model-Powered Embodied Agents: Attack Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2608.16843/) (2026)

## Abstract (arXiv)

Recent advances in embodied Vision-Language Agentic Systems (VLAS), powered by large vision-language models (LVLMs), enable AI systems to perceive and reason over real-world scenes. Within this context, environmental signals such as traffic lights are essential in-band signals that can and should influence agent behavior. However, similar signals could also be crafted to operate as misleading visual injections, overriding user intent and posing security risks. This duality creates a fundamental challenge: agents must respond to legitimate environmental cues while remaining robust to misleading ones. We refer to this tension as trust boundary confusion. To study this behavior, we design a dual-intent dataset and evaluation framework, through which we show that current LVLM-based agents fail to reliably balance this trade-off, either ignoring useful signals or following harmful ones. We systematically evaluate 7 LVLM agents across multiple embodied settings under both structure-based and noise-based visual injections. To address these vulnerabilities, we propose a multi-agent defense framework that separates perception from decision-making to dynamically assess the reliability of visual inputs. Our approach significantly reduces misleading behaviors while preserving correct responses and provides robustness guarantees under adversarial perturbations. The code of the evaluation framework and artifacts are made available at https://anonymous.4open.science/r/Visual-Prompt-Inject.