VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. We present VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search. First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step -- with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies show that all three proposed components consistently improve a strong backbone, while a weaker backbone benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 37 of its 48 references that are in the map. First posted on arXiv 23 Apr 2026.
Builds on
37 of 48 references.
Closest ancestors · 14
Same direction, newest first
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025Scaling Agents for Computer Use2 Oct
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agents4 Jun
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
2 more
Other references · 23
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
- 2024Executable Code Actions Elicit Better LLM Agentscited by 537
- 2023Language Models can Solve Computer Taskscited by 371
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
11 more
- 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
- 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
- 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
- 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 33
- 2023A Zero-Shot Language Agent for Computer Control with Structured Reflectioncited by 28
- 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
- 2025Mano Technical Reportcited by 13
- 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
- 2026Absorbed in Inertia: Activation Analysis for Computer-Use Agentscited by 0
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolutioncited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution1 Oct 2026
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents30 Sep 2026
- 2026Absorbed in Inertia: Activation Analysis for Computer-Use Agents29 Sep 2026
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents
- 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction
- 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
- 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point
- 2026Plover: Steering GUI Agents through Plan-Centric Interaction
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
- 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
- 2026WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments