mlsanityData to 14 Oct 2026

ViMo: A Generative Visual GUI World Model for App Agents

arXiv 2504.13936 ↗First posted 15 Apr 2025cs.HC cs.LG cs.SYcited by 35

On the map: World models → World model security and GUI agents

Abstract (arXiv)

App agents, which autonomously operate mobile Apps through Graphical User Interfaces (GUIs), have gained significant interest in real-world applications. Yet, they often struggle with long-horizon planning, failing to find the optimal actions for complex tasks with longer steps. To address this, world models are used to predict the next GUI observation based on user actions, enabling more effective agent planning. However, existing world models primarily focus on generating only textual descriptions, lacking essential visual details. To fill this gap, we propose ViMo, the first visual world model designed to generate future App observations as images. For the challenge of generating text in image patches, where even minor pixel errors can distort readability, we decompose GUI generation into graphic and text content generation. We propose a novel data representation, the Symbolic Text Representation~(STR) to overlay text content with symbolic placeholders while preserving graphics. With this design, ViMo employs a STR Predictor to predict future GUIs' graphics and a GUI-text Predictor for generating the corresponding text. Moreover, we deploy ViMo to enhance agent-focused tasks by predicting the outcome of different action options. Experiments show ViMo's ability to generate visually plausible and functionally effective GUIs that enable App agents to make more informed decisions.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 44 of its 68 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

44 of 68 references.

References · 44

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  10. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 520
  11. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
32 more
  1. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  2. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  3. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  4. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 259
  5. 2020The Creation and Detection of Deepfakes: A Surveycited by 247
  6. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  7. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  8. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
  9. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  10. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  11. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  12. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  13. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  14. 2024Tree Search for Language Model Agentscited by 151
  15. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 140
  16. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  17. 2020SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 119
  18. 2018Image Generation from Layoutcited by 105
  19. 2024GUI Agents: A Surveycited by 103
  20. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
  21. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  22. 2024TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Controlcited by 45
  23. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Modelcited by 41
  24. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 34
  25. 2017Learning model-based planning from scratchcited by 22
  26. 2023UI Layout Generation with LLMs Guided by UI Grammarcited by 15
  27. 2024Lightweight Neural App Controlcited by 12
  28. 2024LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Neededcited by 10
  29. 2024Few-Shot Image Generation by Conditional Relaxing Diffusion Inversioncited by 7
  30. 2023A Picture is Worth a Thousand Words: Language Models Plan from Pixelscited by 6
  31. 2024On AI-Inspired UI-Designcited by 4
  32. 2023GUILGET: GUI Layout GEneration with Transformercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 31.

35 papers.

Sort:

By area:

  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  2. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  3. 2026Code2World: A GUI World Model via Renderable Code Generationcited by 29
  4. 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
  5. 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
  6. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  7. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
  8. 2026Generative Visual Code Mobile World Modelscited by 13
  9. 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 13
  10. 2026How Mobile World Model Guides GUI Agents?cited by 8
  11. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  12. 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 7
23 more
  1. 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systemscited by 6
  2. 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environmentscited by 5
  3. 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 5
  4. 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamicscited by 5
  5. 2026Anticipatory Planning for Multimodal AI Agentscited by 4
  6. 2026Scaling GUI Agents with Visual State Transitionscited by 4
  7. 2025Beyond Syntax: Action Semantics Learning for App Agentscited by 3
  8. 2026SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Predictioncited by 3
  9. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoningcited by 2
  10. 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
  11. 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 2
  12. 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
  13. 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Modelscited by 1
  14. 2026Software Engineering for and with GUI Agentcited by 1
  15. 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
  16. 2026Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignmentcited by 0
  17. 2026Quo Vadis, World Modeling?cited by 0
  18. 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learningcited by 0
  19. 2026Solaris: Towards Interfaces That Are Generated, Not Codedcited by 0
  20. 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
  21. 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentscited by 0
  22. 2026Latent World Action Model for GUI Agentscited by 0
  23. 2026StateCraft: Augmenting Video World Models with Abstract-State Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MobileDreamer: Generative Sketch World Model for GUI Agent
  2. 2026Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
  3. 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
  4. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  5. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
  6. 2025Computer-Use Agents as Judges for Generative User Interface
  7. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  8. 2026SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis