Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 40 references that are in the map. First posted on arXiv 7 Apr 2026.
Builds on
34 of 40 references.
Closest ancestors · 15
Same direction, newest first
- 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation21 Jul
- 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism27 May
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation5 Feb
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
- 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents18 Oct
- 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
3 more
Other references · 19
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
- 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
7 more
- 2022Generating Sequences by Learning to Self-Correctcited by 271
- 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
- 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
- 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
- 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
Built on by
Citing papers per year: 2026: 14.
14 papers.
Sort:
By area:
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
- 2026What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoningcited by 4
- 2026StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structurecited by 4
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Datacited by 1
- 2026The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agentscited by 1
- 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environmentscited by 1
- 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
- 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentscited by 1
- 2026FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systemscited by 0
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agentscited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution1 Oct 2026
- 2026Absorbed in Inertia: Activation Analysis for Computer-Use Agents29 Sep 2026
- 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agents26 Sep 2026
- 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents15 Sep 2026
- 2026Agents in the Large: Perception-Centered Architecture for Persistent Agents31 Aug 2026
- 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments25 Aug 2026
- 2026The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents6 Aug 2026
- 2026StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure13 Jul 2026
- 2026Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data3 Jun 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems16 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actionscited by 6
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agentscited by 1
- 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environmentscited by 1
- 2026Agents in the Large: Perception-Centered Architecture for Persistent Agentscited by 1
- 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agentscited by 1
- 2026FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systemscited by 0
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agentscited by 0
- 2026Absorbed in Inertia: Activation Analysis for Computer-Use Agentscited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolutioncited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution1 Oct 2026
- 2026Absorbed in Inertia: Activation Analysis for Computer-Use Agents29 Sep 2026
- 2026SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agents26 Sep 2026
- 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents15 Sep 2026
- 2026Agents in the Large: Perception-Centered Architecture for Persistent Agents31 Aug 2026
- 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments25 Aug 2026
- 2026The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents6 Aug 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems16 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
- 2026Android Coach: Improve Online Agentic Training Efficiency with Single State Multiple Actions8 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
- 2026Learning Reliable GUI Agents under Imperfect Priors
- 2026Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
- 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
- 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
- 2026Plover: Steering GUI Agents through Plan-Centric Interaction
- 2025ZeroGUI: Automating Online GUI Learning at Zero Human Cost
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks