mlsanityData to 7 Oct 2026

BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism

arXiv 2505.20660 ↗First posted 27 May 2025cs.CL cs.AIcited by 25

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Graphical User Interface (GUI) agents have gained substantial attention due to their impressive capabilities to complete tasks through multiple interactions within GUI environments. However, existing agents primarily focus on enhancing the accuracy of individual actions and often lack effective mechanisms for detecting and recovering from errors. To address these shortcomings, we propose the BacktrackAgent, a robust framework that incorporates a backtracking mechanism to improve task completion efficiency. BacktrackAgent includes verifier, judger, and reflector components as modules for error detection and recovery, while also applying judgment rewards to further enhance the agent's performance. Additionally, we develop a training dataset specifically designed for the backtracking mechanism, which considers the outcome pages after action executions. Experimental results show that BacktrackAgent has achieved performance improvements in both task success rate and step accuracy on Mobile3M and Auto-UI benchmarks. Our data and code will be released upon acceptance.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 38 of its 53 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

38 of 53 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025ReachAgent: Enhancing Mobile Agent via Page Reaching and Operation5 Feb
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  3. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
  4. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
  5. 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
  6. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents18 Oct
  7. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  8. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents3 Jul
  9. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  10. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  11. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation19 Feb
  12. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
2 more
  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  2. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2022CodeT: Code Generation with Generated Testscited by 369
  8. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  9. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  10. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  11. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  12. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
12 more
  1. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  2. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
  3. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  5. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  6. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  7. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  8. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 46
  9. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  10. 2024Large Action Models: From Inception to Implementationcited by 18
  11. 2024Devil's Advocate: Anticipatory Reflection for LLM Agentscited by 17
  12. 2024Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 21.

25 papers.

Sort:

By area:

  1. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 25
  2. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  3. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 15
  4. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correctioncited by 14
  5. 2025Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Modelscited by 6
  6. 2026GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosiscited by 6
  7. 2026Xiaomi-GUI-0 Technical Reportcited by 5
  8. 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environmentcited by 4
  9. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  10. 2026Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planningcited by 3
  11. 2026CUADebug: Diagnosing and Repairing Computer-Use Agent Failurescited by 3
  12. 2025InfraMind: A Novel Exploration-based GUI Agentic Framework for Mission-critical Industrial Managementcited by 2
13 more
  1. 2026Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agentscited by 2
  2. 2026BEAP-Agent: Backtrackable Execution and Adaptive Planning for GUI Agentscited by 1
  3. 2026Diagnosing and Mitigating Compounding Failures in Agentic Persuasion via Taxonomic Strategy Retrievalcited by 1
  4. 2026Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environmentscited by 1
  5. 2026Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agentscited by 1
  6. 2026Autonomous Data Processing using Meta-Agentscited by 0
  7. 2026From Control to Foresight: Simulation as a New Paradigm for Human-Agent Collaborationcited by 0
  8. 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
  9. 2026VisCritic: Visual State Comparison as Process Reward for GUI Agentscited by 0
  10. 2026Reflection with Action-Induced Visual Differences for Desktop GUI Agentscited by 0
  11. 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimizationcited by 0
  12. 2026BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agentscited by 0
  13. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ScaleTrack: Scaling and back-tracking Automated GUI Agents
  2. 2025Instruction Agent: Enhancing Agent with Expert Demonstration
  3. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
  4. 2025ProBench: Benchmarking GUI Agents with Accurate Process Information
  5. 2026Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification
  6. 2026Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
  7. 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
  8. 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux