ViMo: A Generative Visual GUI World Model for App Agents
On the map: World models → World model security and GUI agents
Abstract (arXiv)
App agents, which autonomously operate mobile Apps through Graphical User Interfaces (GUIs), have gained significant interest in real-world applications. Yet, they often struggle with long-horizon planning, failing to find the optimal actions for complex tasks with longer steps. To address this, world models are used to predict the next GUI observation based on user actions, enabling more effective agent planning. However, existing world models primarily focus on generating only textual descriptions, lacking essential visual details. To fill this gap, we propose ViMo, the first visual world model designed to generate future App observations as images. For the challenge of generating text in image patches, where even minor pixel errors can distort readability, we decompose GUI generation into graphic and text content generation. We propose a novel data representation, the Symbolic Text Representation~(STR) to overlay text content with symbolic placeholders while preserving graphics. With this design, ViMo employs a STR Predictor to predict future GUIs' graphics and a GUI-text Predictor for generating the corresponding text. Moreover, we deploy ViMo to enhance agent-focused tasks by predicting the outcome of different action options. Experiments show ViMo's ability to generate visually plausible and functionally effective GUIs that enable App agents to make more informed decisions.
Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 44 of its 68 references that are in the map. First posted on arXiv 15 Apr 2025.
Builds on
44 of 68 references.
References · 44
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2024GPT-4o System Cardcited by 4,588
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
- 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
- 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 520
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
32 more
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
- 2024Diffusion Models Are Real-Time Game Enginescited by 296
- 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
- 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 259
- 2020The Creation and Detection of Deepfakes: A Surveycited by 247
- 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
- 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
- 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
- 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
- 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
- 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
- 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
- 2024Tree Search for Language Model Agentscited by 151
- 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 140
- 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
- 2020SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognitioncited by 119
- 2018Image Generation from Layoutcited by 105
- 2024GUI Agents: A Surveycited by 103
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
- 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
- 2024TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Controlcited by 45
- 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Modelcited by 41
- 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 34
- 2017Learning model-based planning from scratchcited by 22
- 2023UI Layout Generation with LLMs Guided by UI Grammarcited by 15
- 2024Lightweight Neural App Controlcited by 12
- 2024LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Neededcited by 10
- 2024Few-Shot Image Generation by Conditional Relaxing Diffusion Inversioncited by 7
- 2023A Picture is Worth a Thousand Words: Language Models Plan from Pixelscited by 6
- 2024On AI-Inspired UI-Designcited by 4
- 2023GUILGET: GUI Layout GEneration with Transformercited by 2
Built on by
Citing papers per year: 2025: 4; 2026: 31.
35 papers.
Sort:
By area:
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
- 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
- 2026Code2World: A GUI World Model via Renderable Code Generationcited by 29
- 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
- 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
- 2026Generative Visual Code Mobile World Modelscited by 13
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 13
- 2026How Mobile World Model Guides GUI Agents?cited by 8
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
- 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 7
23 more
- 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systemscited by 6
- 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environmentscited by 5
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 5
- 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamicscited by 5
- 2026Anticipatory Planning for Multimodal AI Agentscited by 4
- 2026Scaling GUI Agents with Visual State Transitionscited by 4
- 2025Beyond Syntax: Action Semantics Learning for App Agentscited by 3
- 2026SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Predictioncited by 3
- 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoningcited by 2
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 2
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Modelscited by 1
- 2026Software Engineering for and with GUI Agentcited by 1
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignmentcited by 0
- 2026Quo Vadis, World Modeling?cited by 0
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learningcited by 0
- 2026Solaris: Towards Interfaces That Are Generated, Not Codedcited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentscited by 0
- 2026Latent World Action Model for GUI Agentscited by 0
- 2026StateCraft: Augmenting Video World Models with Abstract-State Reasoningcited by 0
- 2026StateCraft: Augmenting Video World Models with Abstract-State Reasoning14 Oct 2026
- 2026Latent World Action Model for GUI Agents9 Oct 2026
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents8 Oct 2026
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent1 Oct 2026
- 2026Solaris: Towards Interfaces That Are Generated, Not Coded1 Sep 2026
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments30 Aug 2026
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning27 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026Quo Vadis, World Modeling?3 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction17 Jul 2026
23 more
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application10 Jun 2026
- 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models3 Jun 2026
- 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning2 Jun 2026
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research25 May 2026
- 2026Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment14 May 2026
- 2026How Mobile World Model Guides GUI Agents?11 May 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond24 Apr 2026
- 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation13 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents25 Mar 2026
- 2026Anticipatory Planning for Multimodal AI Agents17 Mar 2026
- 2026WebWorld: A Large-Scale World Model for Web Agent Training16 Feb 2026
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training15 Feb 2026
- 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics11 Feb 2026
- 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb 2026
- 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments9 Feb 2026
- 2026Generative Visual Code Mobile World Models2 Feb 2026
- 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents18 Jan 2026
- 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents16 Dec 2025
- 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems5 Oct 2025
- 2025Beyond Syntax: Action Semantics Learning for App Agents21 Jun 2025
- 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects28 Apr 2025
- 2026Code2World: A GUI World Model via Renderable Code Generationcited by 29
- 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
- 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agentscited by 18
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
- 2026Generative Visual Code Mobile World Modelscited by 13
- 2026How Mobile World Model Guides GUI Agents?cited by 8
- 2026SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Predictioncited by 3
- 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoningcited by 2
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 2
- 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Modelscited by 1
- 2026Quo Vadis, World Modeling?cited by 0
12 most cited of 18 All 18 (JSON)
- 2026StateCraft: Augmenting Video World Models with Abstract-State Reasoning14 Oct 2026
- 2026Latent World Action Model for GUI Agents9 Oct 2026
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents8 Oct 2026
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent1 Oct 2026
- 2026Solaris: Towards Interfaces That Are Generated, Not Coded1 Sep 2026
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments30 Aug 2026
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning27 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026Quo Vadis, World Modeling?3 Aug 2026
- 2026SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction17 Jul 2026
- 2026MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models3 Jun 2026
- 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning2 Jun 2026
12 newest of 18 All 18 (JSON)
- 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Researchcited by 13
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
- 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 7
- 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systemscited by 6
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 5
- 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamicscited by 5
- 2026Scaling GUI Agents with Visual State Transitionscited by 4
- 2025Beyond Syntax: Action Semantics Learning for App Agentscited by 3
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026Software Engineering for and with GUI Agentcited by 1
12 most cited of 13 All 13 (JSON)
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application10 Jun 2026
- 2026MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research25 May 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation13 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents25 Mar 2026
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training15 Feb 2026
- 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics11 Feb 2026
- 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems5 Oct 2025
- 2025Beyond Syntax: Action Semantics Learning for App Agents21 Jun 2025
12 newest of 13 All 13 (JSON)
- 2026Anticipatory Planning for Multimodal AI Agentscited by 4
- 2026Anticipatory Planning for Multimodal AI Agents17 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026MobileDreamer: Generative Sketch World Model for GUI Agent
- 2026Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
- 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
- 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
- 2025Computer-Use Agents as Judges for Generative User Interface
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2026SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis