Code Plans, Diffusion Renders: Open-Ended Generative World Modeling
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
We introduce CoDeR, a new paradigm for world modeling. Unlike existing video world models that implicitly represent world dynamics through visual observations, our system explicitly constructs an executable world with code and employs video generation models for visual realization. Specifically, we coordinate five complementary roles to translate high-level concepts into structured world rules, executable dynamics, and perceptual observations. This design enables textitlong-term memory, textitopen-ended interactions, textitautonomous world evolution, and textitmulti-agent scenarios, where multiple entities can act, interact, and evolve persistently beyond the current observation. Extensive experiments demonstrate that our framework substantially extends the capabilities of existing world models, enabling long-term memory, open-ended interactions, autonomous evolution, and persistent multi-agent dynamics, while achieving state-of-the-art performance across multiple evaluation settings. Code and model weights will be made publicly available. Project Page: hrefhttps://becauseimbatman0.github.io/CoDeRCoDeR.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 98 of its 108 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
98 of 108 references.
Closest ancestors · 53
Same direction, newest first
- 2026Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory30 Aug
- 2026ReWorld: An Interactive World Model with Long-Horizon Memory24 Aug
- 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearance14 Aug
- 2026AlayaWorld: Interactive Long-Horizon World Modeling - Full Technical Report (v1.1)13 Aug
- 2026Population-Scalable Multi-Agent World Modeling9 Aug
- 2026MASS: Multiplayer World Models with Authoritative Shared State6 Aug
- 2026Wonder: Video World Model Done Better28 Jul
- 2026Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers23 Jul
- 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU21 Jul
- 2026Multiplayer Interactive World Models with Representation Autoencoders6 Jul
- 2026MemLearner: Learning to Query Context memory for Video World Models30 Jun
- 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memory16 Jun
38 more
- 2026DreamX-World 1.0: A General-Purpose Interactive World Model15 Jun
- 2026MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold11 Jun
- 2026Latent Spatial Memory for Video World Models8 Jun
- 2026Prisma-World: Camera-Controllable Multi-Agent Video World Model8 Jun
- 2026AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization5 Jun
- 2026MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data1 Jun
- 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players27 May
- 2026Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution25 May
- 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models22 May
- 2026Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models18 May
- 2026ChronoAgentic: A Code-based Multi-Agent World Simulator for Physically Grounded Simulation Construction14 May
- 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Models20 Apr
- 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory10 Apr
- 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models26 Mar
- 2026MosaicMem: Hybrid Spatial Memory for Controllable Video World Models17 Mar
- 2026Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models13 Mar
- 2026LiveWorld: Simulating Out-of-Sight Dynamics in Generative Video World Models7 Mar
- 2026Beyond Pixel Histories: World Models with Persistent 3D State3 Mar
- 2026Solaris: Building a Multiplayer Video World Model in Minecraft25 Feb
- 2026Advancing Open-source World Models28 Jan
- 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos1 Jan
- 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling16 Dec
- 2025RELIC: Interactive Video World Model with Long-Horizon Memory3 Dec
- 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modeling2 Dec
- 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model28 Nov
- 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft3 Oct
- 2025Matrix-game 2.0: An open-source real-time and streaming interactive world model18 Aug
- 2025Yume: An Interactive World Generation Model23 Jul
- 2025Video World Models with Long-term Spatial Memory5 Jun
- 2025DeepVerse: 4D Autoregressive Video Generation as a World Model1 Jun
- 2025VRAG: Learning World Models for Interactive Video Generation28 May
- 2025WorldMem: Long-term Consistent World Simulation with Memory16 Apr
- 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft11 Apr
- 2025WorldModelBench: Judging Video Generation Models As World Models28 Feb
- 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
- 2024GameGen-X: Interactive Open-world Game Video Generation1 Nov
- 2024Diffusion Models Are Real-Time Game Engines27 Aug
- 2024Genie: Generative Interactive Environments23 Feb
50 newest of 53 All 53 (JSON)
Other references · 45
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2024Depth Anything V2cited by 1,422
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
- 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
- 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
- 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
- 2024Navigation World Modelscited by 272
- 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
33 more
- 2025World Simulation with Video Foundation Models for Physical AIcited by 200
- 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
- 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion modelscited by 161
- 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
- 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
- 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 109
- 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
- 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
- 2025Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generationcited by 101
- 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 98
- 2025Aether: Geometric-Aware Unified World Modelingcited by 97
- 2025Yume-1.5: A Text-Controlled Interactive World Generation Modelcited by 84
- 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
- 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 66
- 2026Infinite Worlds with Versatile Interactionscited by 48
- 2026Lyra 2.0: Explorable Generative 3D Worldscited by 44
- 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
- 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
- 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 26
- 2025Dexterous World Modelscited by 18
- 2026Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gesturescited by 16
- 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 15
- 2026EgoSim: Egocentric World Simulator for Embodied Interaction Generationcited by 14
- 2026Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Controlcited by 13
- 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 13
- 2026Egocentric World Model for Photorealistic Hand-Object Interaction Synthesiscited by 12
- 2025From Virtual Games to Real-World Playcited by 9
- 2026CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generationcited by 7
- 2025OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenescited by 4
- 2026Magpie: Real-Time World Renderer for Interactive Gamescited by 3
- 2026ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadowcited by 2
- 2026Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Meshcited by 2
- 2026IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Trainingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Code World Model: Coding Agent as World Brain
- 2026Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
- 2026StateCraft: Augmenting Video World Models with Abstract-State Reasoning
- 2026VisualPatchWorld: Code World Models as Latent Structured Representations for Planning
- 2026ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling
- 2025From Masks to Worlds: A Hitchhiker's Guide to World Models
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
- 2025World-in-World: World Models in a Closed-Loop World