# Invisible to Humans, Triggered by Agents: Stealthy Jailbreak Attacks on Mobile Vision-Language Agents

arXiv: https://arxiv.org/abs/2510.07809 · HTML: https://mlsanity.com/paper/2510.07809/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage](https://mlsanity.com/paper/2409.11295/) (2024)
- [Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks](https://mlsanity.com/paper/2501.11733/) (2025)
- [GUI Agents with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2411.04890/) (2024)
- [InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection](https://mlsanity.com/paper/2501.04575/) (2025)
- [Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents](https://mlsanity.com/paper/2410.13886/) (2024)
- [WIPI: A New Web Threat for LLM-Driven Web Agents](https://mlsanity.com/paper/2402.16965/) (2024)
- [AppAgentX: Evolving GUI Agents as Proficient Smartphone Users](https://mlsanity.com/paper/2503.02268/) (2025)
- [Foundations and Recent Trends in Multimodal Mobile Agents: A Survey](https://mlsanity.com/paper/2411.02006/) (2024)
- [MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments](https://mlsanity.com/paper/2506.01616/) (2025)
- [Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks](https://mlsanity.com/paper/2502.13053/) (2025)
- [ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents](https://mlsanity.com/paper/2410.11872/) (2024)
- [From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents](https://mlsanity.com/paper/2505.12981/) (2025)
- [M2S: Multi-turn to Single-turn jailbreak in Red Teaming for LLMs](https://mlsanity.com/paper/2503.04856/) (2025)

## Built on by (most cited first, 3 of 3)

- [Adversarial Confusion Attack: Disrupting Multimodal Large Language Models](https://mlsanity.com/paper/2511.20494/) (2025)
- [Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks](https://mlsanity.com/paper/2605.15118/) (2026)
- [Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2608.04759/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) empower autonomous mobile agents, yet their security under realistic mobile deployment constraints remains underexplored. While agents are vulnerable to visual prompt injections, stealthily executing such attacks without requiring system-level privileges remains challenging, as existing methods rely on persistent visual manipulations that are noticeable to users. We uncover a consistent discrepancy between human and agent interactions: automated agents generate near-zero contact touch signals. Building on this insight, we propose a new attack paradigm, agent-only perceptual injection, where malicious content is exposed only during agent interactions, while remaining not readily perceived by human users. To accommodate mobile UI constraints and one-shot interaction settings, we introduce HG-IDA*, an efficient one-shot optimization method for constructing jailbreak prompts that evade LVLM safety filters. Experiments demonstrate that our approach induces unauthorized cross-app actions, achieving 82.5% planning and 75.0% execution hijack rates on GPT-4o. Our findings highlight a previously underexplored attack surface in mobile agent systems and underscore the need for defenses that incorporate interaction-level signals.