mlsanityData to 14 Oct 2026

How Mobile World Model Guides GUI Agents?

arXiv 2605.10347 ↗First posted 11 May 2026cs.AI cs.CLcited by 8

On the map: World models → World model security and GUI agents

also in 2 more
Abstract (arXiv)

Recent advances in vision-language models have enabled mobile GUI agents to perceive visual interfaces and execute user instructions, but reliable prediction of action consequences remains critical for long-horizon and high-risk interactions. Existing mobile world models provide either text-based or image-based future states, yet it remains unclear which representation is useful, whether generated rollouts can replace real environments, and how test-time guidance helps agents of different strengths. To answer the above questions, we filter and annotate mobile world-model data, then train world models across four modalities: delta text, full text, diffusion-based images, and renderable code. These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench. Furthermore, by evaluating their downstream utility on AITZ, AndroidControl, and AndroidWorld, we obtain three findings. First, renderable code reconstruction achieves high in-distribution fidelity and provides effective multimodal supervision for data construction, while text-based feedback is more robust for online out-of-distribution (OOD) execution. Second, world-model-generated trajectories can provide transferable interaction experience in the training process and improve agents' end-to-end task performance, although these data do not preserve the original distribution. Last, for overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains, suggesting that world models are more effective as prior perception or training supervision than as universal post-hoc verifiers.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 49 of its 53 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

49 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb
  2. 2026Generative Visual Code Mobile World Models2 Feb
  3. 2026MobileDreamer: Generative Sketch World Model for GUI Agent7 Jan
  4. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents16 Dec
  5. 2025ViMo: A Generative Visual GUI World Model for App Agents15 Apr

Other references · 44

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  8. 2025Qwen-Image Technical Reportcited by 1,044
  9. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  10. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  11. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
32 more
  1. 2025Seed1.5-VL Technical Reportcited by 373
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  3. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  4. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  5. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  6. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  7. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  8. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  9. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  10. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  11. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  12. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 131
  13. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  14. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  15. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 91
  16. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 79
  17. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  18. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  19. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
  20. 2024MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understandingcited by 48
  21. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?cited by 42
  22. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
  23. 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
  24. 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
  25. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  26. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  27. 2026Computer-Using World Modelcited by 16
  28. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  29. 2025STEP: Success-Rate-Aware Trajectory-Efficient Policy Optimizationcited by 12
  30. 2025MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learningcited by 9
  31. 2026CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities Reasoningcited by 5
  32. 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning
  2. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
  3. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  4. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  5. 2026GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models
  6. 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents
  7. 2026Latent World Action Model for GUI Agents
  8. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning