Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
Graphical user interface (GUI) agents powered by multimodal large language models (MLLMs) have shown greater promise for human-interaction. However, due to the high fine-tuning cost, users often rely on open-source GUI agents or APIs offered by AI providers, which introduces a critical but underexplored supply chain threat: backdoor attacks. In this work, we first unveil that MLLM-powered GUI agents naturally expose multiple interaction-level triggers, such as historical steps, environment states, and task progress. Based on this observation, we introduce AgentGhost, an effective and stealthy framework for red-teaming backdoor attacks. Specifically, we first construct composite triggers by combining goal and interaction levels, allowing GUI agents to unintentionally activate backdoors while ensuring task utility. Then, we formulate backdoor injection as a Min-Max optimization problem that uses supervised contrastive learning to maximize the feature difference across sample classes at the representation space, improving flexibility of the backdoor. Meanwhile, it adopts supervised fine-tuning to minimize the discrepancy between backdoor and clean behavior generation, enhancing effectiveness and utility. Extensive evaluations of various agent models in two established mobile benchmarks show that AgentGhost is effective and generic, with attack accuracy that reaches 99.7% on three attack objectives, and shows stealthiness with only 1% utility degradation. Furthermore, we tailor a defense method against AgentGhost that reduces the attack accuracy to 22.1%. Our code is available at textttanonymous.
Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 39 of its 68 references that are in the map. First posted on arXiv 20 May 2025.
Builds on
39 of 68 references.
Closest ancestors · 5
Same direction, newest first
- 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections15 Apr
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
- 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
- 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
- 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
Other references · 34
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 580
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
- 2019A backdoor attack against LSTM-based text classification systemscited by 227
- 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
- 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
22 more
- 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
- 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
- 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
- 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
- 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
- 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
- 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 104
- 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
- 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 87
- 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
- 2024Instruction Backdoor Attacks Against Customized LLMscited by 62
- 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
- 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
- 2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
- 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 46
- 2023Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Reviewcited by 39
- 2023The Philosopher's Stone: Trojaning Plugins of Large Language Modelscited by 37
- 2023VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planningcited by 28
- 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
- 2024Dynamic Planning for LLM-based Graphical User Interface Automationcited by 19
- 2025Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Taskscited by 11
- 2025BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systemscited by 7
Built on by
Citing papers per year: 2025: 4; 2026: 9.
13 papers.
Sort:
By area:
- 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systemscited by 25
- 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 19
- 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
- 2025Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggerscited by 8
- 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
- 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agentscited by 2
- 2026MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Contentcited by 1
- 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystemcited by 0
- 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safetycited by 0
- 2026OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agentscited by 0
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agentscited by 0
1 more
- 2026EvolvAttack: Backdooring self Evolving VLM Agentscited by 0
- 2026EvolvAttack: Backdooring self Evolving VLM Agents11 Oct 2026
- 2026AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents6 Sep 2026
- 2026MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content27 May 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents27 Apr 2026
- 2026ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety21 Apr 2026
- 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems8 Apr 2026
- 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents24 Mar 2026
- 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agents9 Mar 2026
- 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows28 Oct 2025
- 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystem24 Oct 2025
- 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?23 Oct 2025
- 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systemscited by 25
- 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 19
- 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
- 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
- 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agentscited by 2
- 2026MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Contentcited by 1
- 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystemcited by 0
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agentscited by 0
- 2026EvolvAttack: Backdooring self Evolving VLM Agentscited by 0
- 2026EvolvAttack: Backdooring self Evolving VLM Agents11 Oct 2026
- 2026AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents6 Sep 2026
- 2026MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content27 May 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems8 Apr 2026
- 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents24 Mar 2026
- 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agents9 Mar 2026
- 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows28 Oct 2025
- 2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystem24 Oct 2025
- 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?23 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agents
- 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
- 2025Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustness
- 2025AutoBackdoor: Automating Backdoor Attacks via LLM Agents
- 2025Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents
- 2026Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs
- 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain
- 2026ElasticBack: Stealthy Conditional Backdoor in LLM-Agent Skills via Coupled Trigger-Rule Optimization