mlsanityData to 14 Oct 2026

Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents

arXiv 2505.14418 ↗First posted 20 May 2025cs.CLcited by 13

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Graphical user interface (GUI) agents powered by multimodal large language models (MLLMs) have shown greater promise for human-interaction. However, due to the high fine-tuning cost, users often rely on open-source GUI agents or APIs offered by AI providers, which introduces a critical but underexplored supply chain threat: backdoor attacks. In this work, we first unveil that MLLM-powered GUI agents naturally expose multiple interaction-level triggers, such as historical steps, environment states, and task progress. Based on this observation, we introduce AgentGhost, an effective and stealthy framework for red-teaming backdoor attacks. Specifically, we first construct composite triggers by combining goal and interaction levels, allowing GUI agents to unintentionally activate backdoors while ensuring task utility. Then, we formulate backdoor injection as a Min-Max optimization problem that uses supervised contrastive learning to maximize the feature difference across sample classes at the representation space, improving flexibility of the backdoor. Meanwhile, it adopts supervised fine-tuning to minimize the discrepancy between backdoor and clean behavior generation, enhancing effectiveness and utility. Extensive evaluations of various agent models in two established mobile benchmarks show that AgentGhost is effective and generic, with attack accuracy that reaches 99.7% on three attack objectives, and shows stealthiness with only 1% utility degradation. Furthermore, we tailor a defense method against AgentGhost that reduces the attack accuracy to 22.1%. Our code is available at textttanonymous.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 39 of its 68 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

39 of 68 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections15 Apr
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  4. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun
  5. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb

Other references · 34

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  2. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 580
  3. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  4. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  6. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  8. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  9. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  10. 2019A backdoor attack against LSTM-based text classification systemscited by 227
  11. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  12. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
22 more
  1. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  3. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  4. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  5. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  6. 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
  7. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 104
  8. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
  9. 2024VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Modelscited by 87
  10. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  11. 2024Instruction Backdoor Attacks Against Customized LLMscited by 62
  12. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
  13. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
  14. 2023DroidBot-GPT: GPT-powered UI Automation for Androidcited by 47
  15. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Userscited by 46
  16. 2023Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Reviewcited by 39
  17. 2023The Philosopher's Stone: Trojaning Plugins of Large Language Modelscited by 37
  18. 2023VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planningcited by 28
  19. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  20. 2024Dynamic Planning for LLM-based Graphical User Interface Automationcited by 19
  21. 2025Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Taskscited by 11
  22. 2025BLAST: A Stealthy Backdoor Leverage Attack against Cooperative Multi-Agent Deep Reinforcement Learning based Systemscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agents
  2. 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
  3. 2025Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustness
  4. 2025AutoBackdoor: Automating Backdoor Attacks via LLM Agents
  5. 2025Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents
  6. 2026Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs
  7. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain
  8. 2026ElasticBack: Stealthy Conditional Backdoor in LLM-Agent Skills via Coupled Trigger-Rule Optimization