GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks. This gap stems from two limitations: a shortage of high-quality, action-aligned reasoning data, and the direct adoption of generic post-training pipelines that overlook the unique challenges of GUI agents. We identify two fundamental issues in these pipelines: (i) standard SFT with CoT reasoning often hurts grounding, and (ii) step-wise RLVR-tyle training faces partial verifiability, where multiple actions can be correct but only a single demonstrated action is used for verification. This makes offline step-wise metrics weak predictors of online task success. In this work, we present GUI-Libra, a tailored training recipe that addresses these challenges. First, to mitigate the scarcity of action-aligned reasoning data, we introduce a data construction and filtering pipeline and release a curated 81K GUI reasoning dataset. Second, to reconcile reasoning with grounding, we propose action-aware SFT that mixes reasoning-then-action and direct-action data and reweights tokens to emphasize action and grounding. Third, to stabilize RL under partial verifiability, we identify the overlooked importance of KL regularization in RLVR and show that a KL trust region is critical for improving offline-to-online predictability; we further introduce success-adaptive scaling to downweight unreliable negative gradients. Across diverse web and mobile benchmarks, GUI-Libra consistently improves both step-wise accuracy and end-to-end task completion. Our results suggest that carefully designed post-training and data curation can unlock significantly stronger task-solving capabilities without costly online data collection. We release our dataset, code, and models to facilitate further research on data-efficient post-training for reasoning-capable GUI agents.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 52 of its 84 references that are in the map. First posted on arXiv 25 Feb 2026.
Builds on
52 of 84 references.
Closest ancestors · 20
Same direction, newest first
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering22 Nov
- 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents6 Nov
- 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning23 Oct
- 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents30 Sep
- 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
- 2025Phi-Ground Tech Report: Advancing Perception in GUI Grounding31 Jul
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
8 more
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
- 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 32
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2025Group Sequence Policy Optimizationcited by 788
- 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
20 more
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
- 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
- 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
- 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
- 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
- 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2025Agent Learning via Early Experiencecited by 79
- 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
- 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
- 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
- 2023Aligning Large Language Models with Human: A Surveycited by 31
- 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
- 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 24
- 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
- 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
- 2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 7
- 2025PixelWeb: The First Web GUI Dataset with Pixel-Wise Labelscited by 3
Built on by
Citing papers per year: 2026: 14.
14 papers.
Sort:
By area:
- 2026Code as Agent Harnesscited by 40
- 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 10
- 2026How Mobile World Model Guides GUI Agents?cited by 8
- 2026Mem-W: Latent Memory-Native GUI Agentscited by 3
- 2026Learning Agentic Policy from Action Guidancecited by 3
- 2026CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Trainingcited by 2
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Controlcited by 1
- 2026Software Engineering for and with GUI Agentcited by 1
- 2026Learning Generalizable Behaviors for Terminal Agentscited by 1
- 2026SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoningcited by 0
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning27 Aug 2026
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes25 Aug 2026
- 2026Learning Generalizable Behaviors for Terminal Agents23 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoning8 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026Code as Agent Harness18 May 2026
- 2026PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control15 May 2026
- 2026Learning Agentic Policy from Action Guidance12 May 2026
- 2026How Mobile World Model Guides GUI Agents?11 May 2026
- 2026Mem-W: Latent Memory-Native GUI Agents10 May 2026
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis16 Apr 2026
- 2026Code as Agent Harnesscited by 40
- 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 10
- 2026Mem-W: Latent Memory-Native GUI Agentscited by 3
- 2026Learning Agentic Policy from Action Guidancecited by 3
- 2026Software Engineering for and with GUI Agentcited by 1
- 2026Learning Generalizable Behaviors for Terminal Agentscited by 1
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxescited by 0
- 2026BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes25 Aug 2026
- 2026Learning Generalizable Behaviors for Terminal Agents23 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026Code as Agent Harness18 May 2026
- 2026Learning Agentic Policy from Action Guidance12 May 2026
- 2026Mem-W: Latent Memory-Native GUI Agents10 May 2026
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis16 Apr 2026
- 2026WebXSkill: Skill Learning for Autonomous Web Agents14 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents
- 2026MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation
- 2025NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks
- 2026Learning Reliable GUI Agents under Imperfect Priors
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks
- 2026UI-Venus-2 Technical Report