PhoneBuddy: Training Open Models for Agentic Phone Use
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67% after supervised fine-tuning to 40.67% after real-app RL and 45.33% after mixed RL. On AndroidWorld, the same progression rises from 60.3% to 77.2% to 83.2%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 50 of its 55 references that are in the map. First posted on arXiv 22 Jun 2026.
Builds on
50 of 55 references.
Closest ancestors · 27
Same direction, newest first
- 2026Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields9 Jun
- 2026CLI-Anything: Towards Agent-Native Computer Use2 Jun
- 2026OpenComputer: Verifiable Software Worlds for Computer-Use Agents19 May
- 2026How Mobile World Model Guides GUI Agents?11 May
- 2026Gym-Anything: Turn any Software into an Agent Environment7 Apr
- 2026CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning27 Feb
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training15 Feb
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
- 2025Step-GUI Technical Report17 Dec
- 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents28 Oct
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
15 more
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
- 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents19 Aug
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
- 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agents4 Jun
- 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning18 May
- 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents17 May
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
- 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents1 Jul
- 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents12 Jun
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
- 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec
Other references · 23
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
- 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
- 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
- 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
- 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
11 more
- 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
- 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
- 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
- 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 21
- 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
- 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
- 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 11
- 2026Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflowscited by 11
- 2026CocoaBench: Evaluating Unified Digital Agents in the Wildcited by 6
- 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agentscited by 6
- 2026Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agentscited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agentscited by 8
- 2026BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agentscited by 0
- 2026Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agentscited by 0
- 2026From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinkscited by 0
- 2026From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks25 Sep 2026
- 2026Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents24 Sep 2026
- 2026BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents11 Sep 2026
- 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents30 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis
- 2025OpenPhone: Mobile Agentic Foundation Models
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
- 2026AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
- 2026Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decomposition