Temporal UI State Inconsistency in Desktop GUI Agents: Formalizing and Defending Against TOCTOU Attacks on Computer-Use Agents
Not on the map. Why?
Abstract (arXiv)
GUI agents that control desktop computers via screenshot-and-click loops introduce a new class of vulnerability: the observation-to-action gap (mean 6.51 s on real OSWorld workloads) creates a Time-Of-Check, Time-Of-Use (TOCTOU) window during which an unprivileged attacker can manipulate the UI state. We formalize this as a Visual Atomicity Violation and characterize three concrete attack primitives: (A) Notification Overlay Hijack, (B) Window Focus Manipulation, and (C) Web DOM Injection. Primitive B, the closest desktop analog to Android Action Rebinding, achieves 100% action-redirection success rate with zero visual evidence at the observation time. We propose Pre-execution UI State Verification (PUSV), a lightweight three-layer defense that re-verifies the UI state immediately before each action dispatch: masked pixel SSIM at the click target (L1), global screenshot diff (L2a), and X Window snapshot diff (L2b). PUSV achieves 100% Action Interception Rate across 180 adversarial trials (135 Primitive A + 45 Primitive B) with zero false positives and < 0.1 s overhead. Against Primitive C (zero-visual-footprint DOM injection), PUSV reveals a structural blind spot (~0% AIR), motivating future OS+DOM defense-in-depth architectures. No single PUSV layer alone achieves full coverage; different primitives require different detection signals, validating the layered design.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
19 of 33 references.
References · 19
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Visual Instruction Tuningcited by 7,869
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
- 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
- 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
7 more
- 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 124
- 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 111
- 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 45
- 2025EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attackscited by 9
- 2026Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agentscited by 7
- 2026Mind the Gap: Action Rebinding Attacks against Android GUI Agentscited by 6
- 2026Atomicity for Agents: Exposing, Exploiting, and Mitigating TOCTOU Vulnerabilities in Browser-Use Agentscited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AgentRAE: Remote Action Execution through Notification-based Visual Backdoors against Screenshots-based Mobile GUI Agents
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection
- 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents
- 2026MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps
- 2026MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content
- 2025VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
- 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents