mlsanityData to 7 Oct 2026

GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents

arXiv 2504.10458 ↗First posted 14 Apr 2025cs.CV cs.CL cs.HCcited by 240

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

Existing efforts in building Graphical User Interface (GUI) agents largely rely on the training paradigm of supervised fine-tuning on Large Vision-Language Models (LVLMs). However, this approach not only demands extensive amounts of training data but also struggles to effectively understand GUI screenshots and generalize to unseen interfaces. The issue significantly limits its application in real-world scenarios, especially for high-level tasks. Inspired by Reinforcement Fine-Tuning (RFT) in large reasoning models (e.g., DeepSeek-R1), which efficiently enhances the problem-solving capabilities of large language models in real-world settings, we propose name, the first reinforcement learning framework designed to enhance the GUI capabilities of LVLMs in high-level real-world task scenarios, through unified action space rule modeling. By leveraging a small amount of carefully curated high-quality data across multiple platforms (including Windows, Linux, MacOS, Android, and Web) and employing policy optimization algorithms such as Group Relative Policy Optimization (GRPO) to update the model, name achieves superior performance using only 0.02% of the data (3K vs. 13M) compared to previous state-of-the-art methods like OS-Atlas across eight benchmarks spanning three different platforms (mobile, desktop, and web). These results demonstrate the immense potential of reinforcement learning based on unified action space rule modeling in improving the execution capabilities of LVLMs for real-world GUI agent tasks.

Key facts. Cited by 240 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 31 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

23 of 31 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  2. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  3. 2025MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse24 Mar
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  5. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  6. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  7. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun

Other references · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  9. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  10. 2023Cognitive Architectures for Language Agentscited by 422
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  12. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
4 more
  1. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
  2. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  3. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  4. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 120; 2026: 120.

240 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  6. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  7. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  8. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  9. 2025Memp: Exploring Agent Procedural Memorycited by 88
  10. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  11. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learningcited by 79
  12. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
38 more
  1. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  2. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  3. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  4. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  5. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  6. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  7. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  8. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  9. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  10. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  11. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  12. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  13. 2026UI-Venus-1.5 Technical Reportcited by 39
  14. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  15. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  16. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 36
  17. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
  18. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  19. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  20. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
  21. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  22. 2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
  23. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 25
  24. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agentscited by 25
  25. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  26. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 24
  27. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  28. 2025Table-R1: Inference-Time Scaling for Table Reasoningcited by 23
  29. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
  30. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  31. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  32. 2026Reinforcement World Model Learning for LLM-based Agentscited by 21
  33. 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistencycited by 20
  34. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  35. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  36. 2025Auto-scaling Continuous Memory for GUI Agentcited by 19
  37. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  38. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19

50 most cited of 240 All 240 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents
  2. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  3. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model
  4. 2025GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
  5. 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation
  6. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
  7. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
  8. 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents