mlsanityData to 7 Oct 2026

PhoneBuddy: Training Open Models for Agentic Phone Use

arXiv 2606.23049 ↗First posted 22 Jun 2026cs.CL cs.AIcited by 4

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Phones are becoming an important execution surface for general-purpose agents, but training open models for reliable phone use remains difficult because the environment that matters at deployment, real devices running real apps, is slow, stateful, side-effectful, and hard to reset or verify, while scalable mock environments only approximate real behavior. We present PhoneBuddy, a training recipe and open-model line for agentic phone use that combines a real-app environment with a mock-app environment, PhoneWorld, which reconstructs runnable mock apps from real GUI usage structure. PhoneBuddy first builds a shared supervised fine-tuning stage from trajectories collected in both environments, then compares real-app RL against mixed RL across both environments. Across a 150-task human evaluation on real phones spanning apps, mini-apps, and cross-app workflows, task success rate improves from 36.67% after supervised fine-tuning to 40.67% after real-app RL and 45.33% after mixed RL. On AndroidWorld, the same progression rises from 60.3% to 77.2% to 83.2%. These results show that mock-app training is not a replacement for real-app RL, but a complementary source of scalable, resettable, and automatically checked interaction. The gains are strongest on app and mini-app tasks, while long-horizontal cross-app workflows remain an important open challenge.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 50 of its 55 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

50 of 55 references.

Closest ancestors · 27

Same direction, newest first

  1. 2026Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields9 Jun
  2. 2026CLI-Anything: Towards Agent-Native Computer Use2 Jun
  3. 2026OpenComputer: Verifiable Software Worlds for Computer-Use Agents19 May
  4. 2026How Mobile World Model Guides GUI Agents?11 May
  5. 2026Gym-Anything: Turn any Software into an Agent Environment7 Apr
  6. 2026CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoning27 Feb
  7. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training15 Feb
  8. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  9. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
  10. 2025Step-GUI Technical Report17 Dec
  11. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents28 Oct
  12. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
15 more
  1. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  2. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents19 Aug
  3. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
  4. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agents4 Jun
  5. 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning18 May
  6. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents17 May
  7. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  8. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents31 Oct
  9. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  10. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  11. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents1 Jul
  12. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents12 Jun
  13. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  14. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  15. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec

Other references · 23

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  7. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  8. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  9. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  10. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  11. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  12. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
11 more
  1. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  2. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  3. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
  4. 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 21
  5. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  6. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  7. 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 11
  8. 2026Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflowscited by 11
  9. 2026CocoaBench: Evaluating Unified Digital Agents in the Wildcited by 6
  10. 2026Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agentscited by 6
  11. 2026Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agentscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026PalmClaw: A Native On-Device Agent Framework for Mobile Phones
  2. 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis
  3. 2025OpenPhone: Mobile Agentic Foundation Models
  4. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
  5. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  6. 2025FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
  7. 2026AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
  8. 2026Do Multi-Agents Dream of Electric Screens? Achieving Perfect Accuracy on AndroidWorld Through Task Decomposition