How Mobile World Model Guides GUI Agents?
On the map: World models → World model security and GUI agents
also in 2 more
Abstract (arXiv)
Recent advances in vision-language models have enabled mobile GUI agents to perceive visual interfaces and execute user instructions, but reliable prediction of action consequences remains critical for long-horizon and high-risk interactions. Existing mobile world models provide either text-based or image-based future states, yet it remains unclear which representation is useful, whether generated rollouts can replace real environments, and how test-time guidance helps agents of different strengths. To answer the above questions, we filter and annotate mobile world-model data, then train world models across four modalities: delta text, full text, diffusion-based images, and renderable code. These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench. Furthermore, by evaluating their downstream utility on AITZ, AndroidControl, and AndroidWorld, we obtain three findings. First, renderable code reconstruction achieves high in-distribution fidelity and provides effective multimodal supervision for data construction, while text-based feedback is more robust for online out-of-distribution (OOD) execution. Second, world-model-generated trajectories can provide transferable interaction experience in the training process and improve agents' end-to-end task performance, although these data do not preserve the original distribution. Last, for overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains, suggesting that world models are more effective as prior perception or training supervision than as universal post-hoc verifiers.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 49 of its 53 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
49 of 53 references.
Closest ancestors · 5
Same direction, newest first
- 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb
- 2026Generative Visual Code Mobile World Models2 Feb
- 2026MobileDreamer: Generative Sketch World Model for GUI Agent7 Jan
- 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents16 Dec
- 2025ViMo: A Generative Visual GUI World Model for App Agents15 Apr
Other references · 44
- 2023GPT-4 Technical Reportcited by 18,396
- 2025Qwen3 Technical Reportcited by 8,820
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2025Qwen-Image Technical Reportcited by 1,044
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
32 more
- 2025Seed1.5-VL Technical Reportcited by 373
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
- 2024On the Effects of Data Scale on UI Control Agentscited by 194
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
- 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
- 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
- 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 131
- 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
- 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
- 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
- 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
- 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
- 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 48
- 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?cited by 42
- 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
- 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
- 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
- 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
- 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
- 2026Computer-Using World Modelcited by 16
- 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
- 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 12
- 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learningcited by 9
- 2026CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoningcited by 5
- 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learningcited by 2
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026PhoneBuddy: Training Open Models for Agentic Phone Usecited by 4
- 2026Scaling GUI Agents with Visual State Transitionscited by 4
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 2
- 2026PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agentscited by 1
- 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Modelscited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentscited by 0
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents8 Oct 2026
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent1 Oct 2026
- 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models26 Sep 2026
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments30 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026PhoneBuddy: Training Open Models for Agentic Phone Use22 Jun 2026
- 2026PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents28 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning
- 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
- 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
- 2026GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models
- 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents
- 2026Latent World Action Model for GUI Agents
- 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning