ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present ClawGUI, an open-source framework addressing these three gaps within a single harness. ClawGUI-RL provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. ClawGUI-Eval enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8% reproduction against official baselines. ClawGUI-Agent brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, ClawGUI-2B achieves 17.1% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0%.
Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 59 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
45 of 59 references.
Closest ancestors · 27
Same direction, newest first
- 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents15 Feb
- 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb
- 2026UI-Venus-1.5 Technical Report9 Feb
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
- 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning19 Sep
- 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents8 Sep
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents19 Aug
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents25 Jul
- 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment8 Jul
15 more
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025A Survey on (M)LLM-Based GUI Agents27 Mar
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Users4 Mar
- 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use15 Nov
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024On the Effects of Data Scale on UI Control Agents6 Jun
- 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 18
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
- 2024OmniParser for Pure Vision Based GUI Agentcited by 193
- 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
- 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
- 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
6 more
- 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
- 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
- 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
- 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
- 2025UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameterscited by 12
- 2022UFO2: A unified pre-training framework for online and offline speech recognitioncited by 7
Built on by
Citing papers per year: 2026: 13.
13 papers.
Sort:
By area:
- 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 7
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimizationcited by 5
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026Scaling GUI Agents with Visual State Transitionscited by 3
- 2026AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtreescited by 1
- 2026MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Managementcited by 1
- 2026KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skillcited by 1
- 2026Task-Adaptive Rubrics for GUI Reward Modelingcited by 1
- 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 0
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxescited by 0
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
- 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding14 Sep 2026
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes25 Aug 2026
- 2026Task-Adaptive Rubrics for GUI Reward Modeling25 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill14 Jul 2026
- 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents5 Jul 2026
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization18 Jun 2026
- 2026MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management18 Jun 2026
- 2026AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees19 May 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
1 more
- 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 7
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimizationcited by 5
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026Scaling GUI Agents with Visual State Transitionscited by 3
- 2026MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Managementcited by 1
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxescited by 0
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes25 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents5 Jul 2026
- 2026MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization18 Jun 2026
- 2026MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management18 Jun 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026Task-Adaptive Rubrics for GUI Reward Modelingcited by 1
- 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 0
- 2026Task-Adaptive Rubrics for GUI Reward Modeling25 Aug 2026
- 2026Rethinking Agentic Reinforcement Learning In Large Language Models30 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2026JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
- 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
- 2026ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
- 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
- 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents