Code2World: A GUI World Model via Renderable Code Generation
On the map: World models → World model security and GUI agents
also in 1 more
Abstract (arXiv)
Autonomous GUI agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a vision-language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld navigation. The code is available at https://github.com/AMAP-ML/Code2World.
Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in World model security and GUI agents, one of 11 directions in World models. Builds on 50 of its 53 references that are in the map. First posted on arXiv 10 Feb 2026.
Builds on
50 of 53 references.
Closest ancestors · 3
Same direction, newest first
- 2026MobileDreamer: Generative Sketch World Model for GUI Agent7 Jan
- 2025MobileWorldBench: Towards Semantic World Modeling For Mobile Agents16 Dec
- 2025ViMo: A Generative Visual GUI World Model for App Agents15 Apr
Other references · 47
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024GPT-4o System Cardcited by 4,467
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
35 more
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
- 2024On the Effects of Data Scale on UI Control Agentscited by 187
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 166
- 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
- 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
- 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
- 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
- 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
- 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
- 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
- 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
- 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
- 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
- 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
- 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 22
- 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
- 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
- 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulationcited by 18
- 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 17
- 2025LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Trainingcited by 17
- 2025Grounding Computer Use Agents on Human Demonstrationscited by 16
- 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
- 2023Boosting GUI Prototyping with Diffusion Modelscited by 10
- 2025Computer-Use Agents as Judges for Generative User Interfacecited by 8
- 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 7
- 2025GUI-Rise: Structured Reasoning and History Summarization for GUI Navigationcited by 3
- 2026Urban Socio-Semantic Segmentation with Vision-Language Reasoningcited by 2
Built on by
Citing papers per year: 2026: 26.
26 papers.
Sort:
By area:
- 2026Code as Agent Harnesscited by 40
- 2026Qwen-AgentWorld: Language World Models for General Agentscited by 22
- 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
- 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 13
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Taskscited by 9
- 2026How Mobile World Model Guides GUI Agents?cited by 8
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
- 2026PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environmentscited by 5
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 3
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026Learning Agentic Policy from Action Guidancecited by 3
14 more
- 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 3
- 2026Scaling GUI Agents with Visual State Transitionscited by 3
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 1
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 1
- 2026Object-Centric Environment Modeling for Agentic Taskscited by 0
- 2026Quo Vadis, World Modeling?cited by 0
- 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applicationscited by 0
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learningcited by 0
- 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Modelscited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agentscited by 0
- 2026Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents8 Oct 2026
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent1 Oct 2026
- 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models26 Sep 2026
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments30 Aug 2026
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering28 Aug 2026
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning27 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026Quo Vadis, World Modeling?3 Aug 2026
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks3 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026Object-Centric Environment Modeling for Agentic Tasks3 Jul 2026
- 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications28 Jun 2026
14 more
- 2026Qwen-AgentWorld: Language World Models for General Agents23 Jun 2026
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application10 Jun 2026
- 2026A History-Aware Visually Grounded Critic for Computer Use Agents9 Jun 2026
- 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agents8 Jun 2026
- 2026PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environments29 May 2026
- 2026Code as Agent Harness18 May 2026
- 2026Learning Agentic Policy from Action Guidance12 May 2026
- 2026How Mobile World Model Guides GUI Agents?11 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond24 Apr 2026
- 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents13 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents25 Mar 2026
- 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models23 Mar 2026
- 2026Code as Agent Harnesscited by 40
- 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 13
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Taskscited by 9
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
- 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agentscited by 3
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026Learning Agentic Policy from Action Guidancecited by 3
- 2026Scaling GUI Agents with Visual State Transitionscited by 3
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injectioncited by 1
- 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 1
- 2026Object-Centric Environment Modeling for Agentic Taskscited by 0
12 most cited of 13 All 13 (JSON)
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering28 Aug 2026
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks3 Aug 2026
- 2026Scaling GUI Agents with Visual State Transitions27 Jul 2026
- 2026Object-Centric Environment Modeling for Agentic Tasks3 Jul 2026
- 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications28 Jun 2026
- 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application10 Jun 2026
- 2026A History-Aware Visually Grounded Critic for Computer Use Agents9 Jun 2026
- 2026Code as Agent Harness18 May 2026
- 2026Learning Agentic Policy from Action Guidance12 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents13 Apr 2026
- 2026Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection9 Apr 2026
12 newest of 13 All 13 (JSON)
- 2026Qwen-AgentWorld: Language World Models for General Agentscited by 22
- 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
- 2026How Mobile World Model Guides GUI Agents?cited by 8
- 2026PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environmentscited by 5
- 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 3
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environmentscited by 1
- 2026Quo Vadis, World Modeling?cited by 0
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learningcited by 0
- 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Modelscited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
- 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agent1 Oct 2026
- 2026The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models26 Sep 2026
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments30 Aug 2026
- 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning27 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026Quo Vadis, World Modeling?3 Aug 2026
- 2026Qwen-AgentWorld: Language World Models for General Agents23 Jun 2026
- 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agents8 Jun 2026
- 2026PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environments29 May 2026
- 2026How Mobile World Model Guides GUI Agents?11 May 2026
- 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond24 Apr 2026
- 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models23 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026Generative Visual Code Mobile World Models
- 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents
- 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
- 2026Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
- 2026AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
- 2026Code2Worlds: Empowering Coding LLMs for 4D World Generation
- 2026Solaris: Towards Interfaces That Are Generated, Not Coded
- 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation