"Allow" to Achieve, Over-Privileged Inadvertently: The Unintended Cost of Task-Completion-Driven Pop-up Decisions in Mobile GUI Agents
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Mobile GUI agents routinely encounter system permission dialogs during task execution, yet their ability to grant only permissions that are necessary for the delegated task remains largely unexamined. We present a systematic study of this capability, which we term Permission Literacy. We construct a four-level permission framework based on task relevance and privacy risk and validate the evaluated scenarios with three independent experts in GUI-agent safety. We inject Android-style permission popups into real GUI tasks and evaluate four frontier multimodal large language models using synchronized annotated screenshots and UI-tree hierarchies, making the requester, permission, justification, and available actions accessible to the agent. Beyond the main study, we conduct controlled interventions that separately vary task context and agent-visible requester identity. Under the same Calendar task, changing only the requester from Calendar to PiMusic reduces grants from 26/32 to 0/32, revealing a strong but task-conditioned App-Trust Bias. Holding a popup fixed while changing task context also substantially changes authorization decisions, revealing a systematic Task-Prior Override. Prompt interventions can reduce unnecessary grants, but their effectiveness is inconsistent across models and may come at the cost of suppressing legitimate grants. These results suggest that separating task execution from permission authorization is a promising design direction for future work.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 33 references that are in the map. First posted on arXiv 5 Aug 2026.
Builds on
24 of 33 references.
Closest ancestors · 4
Same direction, newest first
- 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
- 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
- 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 20
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
8 more
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
- 2023Model evaluation for extreme riskscited by 177
- 2024LLM Agents can Autonomously Hack Websitescited by 109
- 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 54
- 2025Towards Automating Data Access Permissions in AI Agentscited by 21
- 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 16
- 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Do Phone-Use Agents Respect Your Privacy?
- 2025Who Grants the Agent Power? Defending Against Instruction Injection via Task-Centric Access Control
- 2026Janus: a Playground for User-Involved Agentic Permission Management
- 2025Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents
- 2025Towards Trustworthy GUI Agents: A Survey
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2026Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
- 2026PriMobiBench: Characterizing Visual Privacy Leakage in VLM-Driven Mobile GUI Agents