Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
On the map: LLM agents and tool use → GUI and mobile agents
also in 1 more
Abstract (arXiv)
Trustworthy deployment of GUI agents in ubiquitous computing settings requires alignment that survives dynamic interaction and precise threat conditions, not just single-turn refusal of explicit harmful requests. We argue that prompt-level alignment, the dominant lightweight defense in current mobile agents, is a local phenomenon: it works reliably only in the narrow evaluation slice where it is typically measured, namely single-turn, explicitly-verbalized intent, and degrades systematically along two axes that any real user can traverse. Using a paired diagnostic on three frontier GUI agents, screen-grounded, user-side persuasion, with no environment injection, we show that a one-line guardrail achieves large single-shot ASR reductions, up to roughly 40 points, at near-zero over-refusal cost. Nevertheless, moving from independent probes to four-turn escalation chains raises guarded ASR by approximately 20 points on every model. Relative to the neutral baselines, this increase reflects substantial guardrail erosion for Qwen but a largely defense-orthogonal dynamic risk for Claude and GPT. The sign of the salience gap flips under the guardrail: concealed requests are not systematically more successful than explicit ones without a guardrail, but are more successful with one, indicating that the defense engages primarily when intent is named. Static single-turn ASR therefore overstates deployed robustness by a systematic and predictable margin.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 39 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
23 of 39 references.
Closest ancestors · 4
Same direction, newest first
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control23 Oct
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 19
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
- 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
7 more
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
- 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
- 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
- 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
- 2024Attacking Vision-Language Computer Agents via Pop-upscited by 116
- 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
- 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Faithful Mobile GUI Agents with Guided Advantage Estimator
- 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents
- 2025Towards Trustworthy GUI Agents: A Survey
- 2026ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices
- 2026The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents
- 2026Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization
- 2026CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation
- 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents