# When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents

arXiv: https://arxiv.org/abs/2609.16732 · HTML: https://mlsanity.com/paper/2609.16732/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 30)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Attacking Vision-Language Computer Agents via Pop-ups](https://mlsanity.com/paper/2411.02391/) (2024)
- [DroidBot-GPT: GPT-powered UI Automation for Android](https://mlsanity.com/paper/2304.07061/) (2023)
- [AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.01391/) (2025)
- [Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment](https://mlsanity.com/paper/2503.15937/) (2025)
- [Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing](https://mlsanity.com/paper/2212.04732/) (2022)
- [GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent](https://mlsanity.com/paper/2505.16827/) (2025)
- [AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation](https://mlsanity.com/paper/2412.18116/) (2024)
- [Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing](https://mlsanity.com/paper/2311.08649/) (2023)
- [Improved GUI Grounding via Iterative Narrowing](https://mlsanity.com/paper/2411.13591/) (2024)
- [SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents](https://mlsanity.com/paper/2507.00841/) (2025)
- [GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?](https://mlsanity.com/paper/2510.20333/) (2025)
- [From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents](https://mlsanity.com/paper/2505.12981/) (2025)
- [GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents](https://mlsanity.com/paper/2601.09770/) (2026)
- [Mind the Gap: Action Rebinding Attacks against Android GUI Agents](https://mlsanity.com/paper/2601.12349/) (2026)
- [GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph](https://mlsanity.com/paper/2601.17418/) (2026)
- [Mobile GUI Agents under Real-world Threats: Are We There Yet?](https://mlsanity.com/paper/2507.04227/) (2025)
- [Analyzing Use of High Privileges on Android: An Empirical Case Study of Screenshot and Screen Recording Applications](https://mlsanity.com/paper/1804.04605/) (2018)
- [AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents](https://mlsanity.com/paper/2603.23007/) (2026)
- [AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents](https://mlsanity.com/paper/2604.20279/) (2026)
- [MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content](https://mlsanity.com/paper/2605.28116/) (2026)

## Abstract (arXiv)

Mobile agents are increasingly capable of autonomously interacting with mobile applications and performing consequential actions on behalf of users. Effective human oversight of such agents relies on a basic premise: users and agents observe consistent information from the same interface. We show that this premise can be systematically violated. Users perceive mobile interfaces through physical displays and the human visual system, making their observations subject to occlusion and luminance contrast limitations. In contrast, agents consume digital screenshots that may retain such content and accessibility representations that expose nonvisual widget metadata. The same UI state can therefore present materially different information to users and agents, a mismatch we term human-agent UI desynchronization. We investigate whether a repackaged clone of a legitimate APK can exploit this desynchronization to steer an agent toward attacker-designated actions, while remaining fully functional and behaviorally consistent with the original application for human users. We demonstrate that this threat is feasible: perturbations embedded before deployment can induce such deviations without access to runtime user instructions, agent detection or online adaptation. To systematically expose and evaluate this threat, we develop an automated framework that constructs user runtime instruction-agnostic UI desynchronization attacks and realizes them in deployable APKs. We conduct static and dynamic evaluations across five mobile-agent frameworks and three backbone models on 546 tasks involving various applications, achieving average misleading rates of 77.9% and 66.9%, respectively. A complementary questionnaire-based study with 186 participants finds that the visual perturbations used in our attacks are difficult for human users to notice.