mlsanityData to 7 Oct 2026

Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

arXiv 2605.29447 ↗First posted 28 May 2026cs.CV cs.CLcited by 2

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains 1,216 executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates 800k high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a 47.4% success rate and a 33.8% All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 50 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

43 of 50 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents19 Mar
  2. 2025Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation27 Nov
  3. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
  4. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  5. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  6. 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies17 Jun
  7. 2025Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System10 Jun
  8. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Cost29 May
  9. 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism27 May
  10. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay22 May
  11. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
  12. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
10 more
  1. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point12 Feb
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  3. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  4. 2024GUI Agents: A Survey18 Dec
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  6. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  7. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  8. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  10. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan

Other references · 21

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2025Qwen3-VL Technical Reportcited by 2,534
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  8. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  9. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  10. 2025Seed1.5-VL Technical Reportcited by 367
  11. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  12. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
9 more
  1. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  2. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  3. 2025Where LLM Agents Fail and How They can Learn From Failurescited by 118
  4. 2025WebExplorer: Explore and Evolve for Training Long-Horizon Web Agentscited by 86
  5. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  6. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  7. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  8. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
  9. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Themcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models
  2. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction
  3. 2025OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Models
  4. 2026GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions
  5. 2026Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errors
  6. 2026TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
  7. 2025D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies
  8. 2026Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks