AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
The recent progress of large language model agents has opened new possibilities for automating tasks through graphical user interfaces (GUIs), especially in mobile environments where intelligent interaction can greatly enhance usability. However, practical deployment of such agents remains constrained by several key challenges. Existing training data is often noisy and lack semantic diversity, which hinders the learning of precise grounding and planning. Models trained purely by imitation tend to overfit to seen interface patterns and fail to generalize in unfamiliar scenarios. Moreover, most prior work focuses on English interfaces while overlooks the growing diversity of non-English applications such as those in the Chinese mobile ecosystem. In this work, we present AgentCPM-GUI, an 8B-parameter GUI agent built for robust and efficient on-device GUI interaction. Our training pipeline includes grounding-aware pre-training to enhance perception, supervised fine-tuning on high-quality Chinese and English trajectories to imitate human-like actions, and reinforcement fine-tuning with GRPO to improve reasoning capability. We also introduce a compact action space that reduces output length and supports low-latency execution on mobile devices. AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks and a new Chinese GUI benchmark called CAGUI, reaching 96.9% Type-Match and 91.3% Exact-Match. To facilitate reproducibility and further research, we publicly release all code, model checkpoint, and evaluation data.
Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 52 of its 63 references that are in the map. First posted on arXiv 2 Jun 2025.
Builds on
52 of 63 references.
Closest ancestors · 21
Same direction, newest first
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
- 2024GUI Agents: A Survey18 Dec
- 2024Large Language Model-Brained GUI Agents: A Survey27 Nov
- 2024GUI Agents with Foundation Models: A Comprehensive Survey7 Nov
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents18 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024AppAgent v2: Advanced Agent for Flexible Mobile Interactions5 Aug
9 more
- 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents3 Jul
- 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
- 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
- 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
- 2023You Only Look at Screens: Multimodal Chain-of-Action Agents20 Sep
Other references · 31
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2024GPT-4o System Cardcited by 4,588
- 2023A Survey of Large Language Modelscited by 2,576
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
19 more
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024Understanding the planning of LLM agents: A surveycited by 421
- 2023Language Models can Solve Computer Taskscited by 372
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
- 2023Tool Learning with Foundation Modelscited by 302
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
- 2024OmniParser for Pure Vision Based GUI Agentcited by 198
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
- 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
- 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
- 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
- 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 74
- 2024Octopus v2: On-device language model for super agentcited by 47
- 2024Octopus: On-device language model for function calling of software APIscited by 27
- 2025AppVLM: A Lightweight Vision Language Model for Online App Controlcited by 15
Built on by
Citing papers per year: 2025: 17; 2026: 27.
44 papers.
Sort:
By area:
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
- 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
- 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
- 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
- 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
- 2026Generative Visual Code Mobile World Modelscited by 13
32 more
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 13
- 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
- 2025Reinforcement Learning for Large Model: A Surveycited by 9
- 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
- 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimizationcited by 7
- 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 7
- 2025InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuningcited by 6
- 2026Mind the Gap: Action Rebinding Attacks against Android GUI Agentscited by 6
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 6
- 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Refluxcited by 5
- 2025See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Togglescited by 4
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
- 2026SecAgent: Efficient Mobile GUI Agent with Semantic Contextcited by 4
- 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environmentscited by 4
- 2026Scaling GUI Agents with Visual State Transitionscited by 4
- 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
- 2026Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agentscited by 3
- 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agentscited by 3
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2025SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsingcited by 2
- 2025AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agentscited by 2
- 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraintscited by 2
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026Faithful Mobile GUI Agents with Guided Advantage Estimatorcited by 1
- 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigationcited by 1
- 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phonescited by 1
- 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesiscited by 0
- 2026M2-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Miningcited by 0
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agentscited by 0
- 2026PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrievalcited by 0
- 2026Learning Reliable GUI Agents under Imperfect Priorscited by 0
- 2026Learning Reliable GUI Agents under Imperfect Priors30 Sep 2026
- 2026PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval29 Sep 2026
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents15 Sep 2026
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones14 Jul 2026
- 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation26 May 2026
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research25 May 2026
- 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining14 May 2026
- 2026Faithful Mobile GUI Agents with Guided Advantage Estimator2 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
32 more
- 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents19 Mar 2026
- 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents16 Mar 2026
- 2026SecAgent: Efficient Mobile GUI Agent with Semantic Context9 Mar 2026
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices25 Feb 2026
- 2026M2-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Mining5 Feb 2026
- 2026Generative Visual Code Mobile World Models2 Feb 2026
- 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments30 Jan 2026
- 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints30 Jan 2026
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution28 Jan 2026
- 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis26 Jan 2026
- 2026MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux19 Jan 2026
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents18 Jan 2026
- 2026Mind the Gap: Action Rebinding Attacks against Android GUI Agents18 Jan 2026
- 2026Compress to Focus: Efficient Coordinate Compression for Policy Optimization in Multi-Turn GUI Agents14 Jan 2026
- 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization4 Jan 2026
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec 2025
- 2025AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents24 Dec 2025
- 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks18 Dec 2025
- 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance17 Nov 2025
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents2 Oct 2025
- 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbations1 Oct 2025
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks28 Sep 2025
- 2025See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles17 Sep 2025
- 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning15 Sep 2025
- 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep 2025
- 2025SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing5 Sep 2025
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep 2025
- 2025InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning27 Aug 2025
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug 2025
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug 2025
- 2025Reinforcement Learning for Large Model: A Survey11 Aug 2025
- 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning19 Jul 2025
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
- 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
- 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 16
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 13
- 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
- 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Taskscited by 7
- 2026Mind the Gap: Action Rebinding Attacks against Android GUI Agentscited by 6
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 6
12 most cited of 27 All 27 (JSON)
- 2026PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval29 Sep 2026
- 2026When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents15 Sep 2026
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones14 Jul 2026
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research25 May 2026
- 2026Faithful Mobile GUI Agents with Guided Advantage Estimator2 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
- 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents19 Mar 2026
- 2026SecAgent: Efficient Mobile GUI Agent with Semantic Context9 Mar 2026
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices25 Feb 2026
12 newest of 27 All 27 (JSON)
- 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
- 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimizationcited by 7
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
- 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigationcited by 1
- 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation26 May 2026
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents18 Jan 2026
- 2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization4 Jan 2026
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents2 Oct 2025
- 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep 2025
- 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 7
- 2025InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuningcited by 6
- 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agentscited by 3
- 2025SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsingcited by 2
- 2026Learning Reliable GUI Agents under Imperfect Priorscited by 0
- 2026Learning Reliable GUI Agents under Imperfect Priors30 Sep 2026
- 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining14 May 2026
- 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents16 Mar 2026
- 2025SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing5 Sep 2025
- 2025InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning27 Aug 2025
- 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning15 Sep 2025
- 2025Reinforcement Learning for Large Model: A Survey11 Aug 2025
- 2026Generative Visual Code Mobile World Modelscited by 13
- 2026Generative Visual Code Mobile World Models2 Feb 2026
- 2025See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Togglescited by 4
- 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
- 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraintscited by 2
- 2026Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints30 Jan 2026
- 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbations1 Oct 2025
- 2025See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles17 Sep 2025
Similar, not cited
Close in wording, no citation either way
- 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
- 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
- 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents
- 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning
- 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects
- 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents