mlsanityData to 7 Oct 2026

Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

arXiv 2604.05477 ↗First posted 7 Apr 2026cs.CLcited by 14

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 40 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

34 of 40 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation21 Jul
  2. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism27 May
  3. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
  4. 2025ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation5 Feb
  5. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  6. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
  7. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
  8. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  9. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
  10. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  11. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents18 Oct
  12. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
3 more
  1. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 19

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  11. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  12. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
7 more
  1. 2022Generating Sequences by Learning to Self-Correctcited by 271
  2. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  3. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  5. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
  6. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  7. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
  2. 2026Learning Reliable GUI Agents under Imperfect Priors
  3. 2026Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
  4. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
  5. 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
  6. 2026Plover: Steering GUI Agents through Plan-Centric Interaction
  7. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Cost
  8. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks