mlsanityData to 7 Oct 2026

PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents

arXiv 2605.29486 ↗First posted 28 May 2026cs.CL cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Real applications provide the training setting closest to phone-agent deployment, but are difficult to reset, scale safely, and verify programmatically. Static screenshots and interaction trajectories preserve realistic evidence but cannot generate new experience. We introduce PhoneWorld, a trace-grounded framework that converts such evidence into runnable, resettable, and verifiable Android environments. PhoneWorld induces a usage-weighted interaction skeleton from observed pages, transitions, and state-changing operations; translates it into a behavior-grounded app specification; realizes the specification through an autonomous build--inspect--repair loop; and synthesizes executable tasks with programmatic verifiers. The resulting suite spans 34 consumer-facing apps across 16 domains and supports an audited online benchmark, verified trajectory generation, and online RL through common reset and verification interfaces. Evaluations with diverse general and open-source GUI agents show that PhoneWorld supports reliable end-to-end online interaction and exposes capabilities complementary to AndroidWorld. Controlled SFT experiments further show that PhoneWorld trajectories complement AndroidWorld supervision, transfer across online and offline benchmarks, and become more effective as data volume and app coverage increase. Under a matched RL budget, combining PhoneWorld mock-app rollouts with real-app rollouts improves performance over real-app RL alone on both real-phone tasks and AndroidWorld. Together, these results demonstrate that trace-grounded executable abstraction can bridge realistic mobile behavior and scalable agent learning, turning limited real-app evidence into a growing supply of controllable and verifiable environments for training and evaluation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 38 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

35 of 38 references.

References · 35

  1. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  3. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  4. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  5. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  6. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  7. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  8. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  9. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  10. 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
  11. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  12. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
23 more
  1. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  2. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environmentscited by 69
  3. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  4. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  5. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 51
  6. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  7. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  8. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  9. 2026UI-Venus-1.5 Technical Reportcited by 39
  10. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
  11. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  12. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  13. 2025A3: Android Agent Arena for Mobile GUI Agents with Essential-State Procedural Evaluationcited by 17
  14. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  15. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  16. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 16
  17. 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 11
  18. 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learningcited by 9
  19. 2026How Mobile World Model Guides GUI Agents?cited by 8
  20. 2026AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applicationscited by 8
  21. 2026CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoningcited by 5
  22. 2025GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learningcited by 4
  23. 2026MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environmentcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.