Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Graphical user interface (GUI) agents now execute workflows across real applications, but when execution drifts, users must determine what changed, what should have remained untouched, and what can still be recovered. Plan-centric interfaces leave users supervising procedure, but users live with changes to application state. We introduce task-state governance, an interaction paradigm that represents a task as an editable contract linking desired outcomes, facts that must be preserved, and observed application state. We instantiate this paradigm in TaskVM, which projects live state into editable task controls, executes user edits as genuine interface actions, verifies outcomes through independent observation, and marks each action as reversible, compensatable, or irreversible. Across an automated contract benchmark, a controlled user study, and an open live-model exploration, the paradigm strengthened contract compliance and users' recovery control and trust, while execution outcomes were held constant. Task contracts open a design space for governable agent interfaces.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 75 references that are in the map. First posted on arXiv 14 Oct 2026.
Builds on
25 of 75 references.
Closest ancestors · 8
Same direction, newest first
- 2026Plover: Steering GUI Agents through Plan-Centric Interaction16 Jul
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research25 May
- 2025From Imperative to Declarative: Towards LLM-friendly OS Interfaces for Boosted Computer-Use Agents6 Oct
- 2025VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification24 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction14 May
Other references · 17
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
- 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 43
- 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 39
- 2024DynaVis: Dynamically Synthesized UI Widgets for Visualization Editingcited by 37
- 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 36
- 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 27
- 2025Morae: Proactively Pausing UI Agents for User Choicescited by 19
5 more
- 2026Generative UI: LLMs are Effective UI Generatorscited by 8
- 2025Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpagescited by 5
- 2025DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfacescited by 4
- 2026Macaron-A2UI: A Model for Generative UI in Personal Agentscited by 2
- 2026Software as Content: Dynamic Applications as the Human-Agent Interaction Layercited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Towards Trustworthy GUI Agents: A Survey
- 2026JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
- 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
- 2026Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents
- 2026Software Engineering for and with GUI Agent
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
- 2026LegacyWorld: Atomicity-Aware Evaluation of GUI Agents for Legacy Workflows
- 2026GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents