mlsanityData to 14 Oct 2026

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning

arXiv 2506.01391 ↗First posted 2 Jun 2025cs.AI cs.CL cs.CVcited by 44

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

The recent progress of large language model agents has opened new possibilities for automating tasks through graphical user interfaces (GUIs), especially in mobile environments where intelligent interaction can greatly enhance usability. However, practical deployment of such agents remains constrained by several key challenges. Existing training data is often noisy and lack semantic diversity, which hinders the learning of precise grounding and planning. Models trained purely by imitation tend to overfit to seen interface patterns and fail to generalize in unfamiliar scenarios. Moreover, most prior work focuses on English interfaces while overlooks the growing diversity of non-English applications such as those in the Chinese mobile ecosystem. In this work, we present AgentCPM-GUI, an 8B-parameter GUI agent built for robust and efficient on-device GUI interaction. Our training pipeline includes grounding-aware pre-training to enhance perception, supervised fine-tuning on high-quality Chinese and English trajectories to imitate human-like actions, and reinforcement fine-tuning with GRPO to improve reasoning capability. We also introduce a compact action space that reduces output length and supports low-latency execution on mobile devices. AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks and a new Chinese GUI benchmark called CAGUI, reaching 96.9% Type-Match and 91.3% Exact-Match. To facilitate reproducibility and further research, we publicly release all code, model checkpoint, and evaluation data.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 52 of its 63 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

52 of 63 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
  2. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  4. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  5. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  6. 2024GUI Agents: A Survey18 Dec
  7. 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
  8. 2024GUI Agents with Foundation Models: A Comprehensive Survey7 Nov
  9. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  10. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents18 Oct
  11. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  12. 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions5 Aug
9 more
  1. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents3 Jul
  2. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  3. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  4. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  5. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  6. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  7. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
  8. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  9. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  11. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
19 more
  1. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  3. 2024Understanding the planning of LLM agents: A surveycited by 421
  4. 2023Language Models can Solve Computer Taskscited by 372
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  6. 2023Tool Learning with Foundation Modelscited by 302
  7. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  8. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  9. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  10. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  12. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
  13. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  14. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  15. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
  16. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 74
  17. 2024Octopus v2: On-device language model for super agentcited by 47
  18. 2024Octopus: On-device language model for function calling of software APIscited by 27
  19. 2025AppVLM: A Lightweight Vision Language Model for Online App Controlcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 27.

44 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
  5. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  7. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  8. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
  9. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  10. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
  11. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  12. 2026Generative Visual Code Mobile World Modelscited by 13
32 more
  1. 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 13
  2. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  3. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  4. 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
  5. 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimizationcited by 7
  6. 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 7
  7. 2025InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuningcited by 6
  8. 2026Mind the Gap: Action Rebinding Attacks against Android GUI Agentscited by 6
  9. 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 6
  10. 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Refluxcited by 5
  11. 2025See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Togglescited by 4
  12. 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
  13. 2026SecAgent: Efficient Mobile GUI Agent with Semantic Contextcited by 4
  14. 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environmentscited by 4
  15. 2026Scaling GUI Agents with Visual State Transitionscited by 4
  16. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
  17. 2026Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agentscited by 3
  18. 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agentscited by 3
  19. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  20. 2025SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsingcited by 2
  21. 2025AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agentscited by 2
  22. 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraintscited by 2
  23. 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
  24. 2026Faithful Mobile GUI Agents with Guided Advantage Estimatorcited by 1
  25. 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigationcited by 1
  26. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phonescited by 1
  27. 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesiscited by 0
  28. 2026M2-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Miningcited by 0
  29. 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
  30. 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agentscited by 0
  31. 2026PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrievalcited by 0
  32. 2026Learning Reliable GUI Agents under Imperfect Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
  2. 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
  3. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents
  4. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  5. 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning
  6. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects
  7. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation
  8. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents