mlsanityData to 7 Oct 2026

GameFactory: Creating New Games with Generative Interactive Videos

arXiv 2501.08325 ↗First posted 14 Jan 2025cs.CVcited by 124

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

Generative videos have the potential to revolutionize game development by autonomously creating new content. In this paper, we present GameFactory, a framework for action-controlled scene-generalizable game video generation. We first address the fundamental challenge of action controllability by introducing GF-Minecraft, an action-annotated game video dataset without human bias, and developing an action control module that enables precise control over both keyboard and mouse inputs. We further extend to support autoregressive generation for unlimited-length interactive videos. More importantly, GameFactory tackles the critical challenge of scene-generalizable action control, which most existing methods fail to address. To enable the creation of entirely new and diverse games beyond fixed styles and scenes, we leverage the open-domain generative priors from pre-trained video diffusion models. To bridge the domain gap between open-domain priors and small-scale game datasets, we propose a multi-phase training strategy with a domain adapter that decouples game style learning from action control. This decoupling ensures that action control learning is no longer bound to specific game styles, thereby achieving scene-generalizable action control. Experimental results demonstrate that GameFactory effectively generates open-domain action-controllable game videos, representing a significant step forward in AI-driven game generation.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

46 of 53 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Open-Sora: Democratizing Efficient Video Production for All29 Dec
  2. 2024GameGen-X: Interactive Open-world Game Video Generation1 Nov
  3. 2024WorldSimBench: Towards Video Generation Models as World Simulators23 Oct
  4. 2024Diffusion Models Are Real-Time Game Engines27 Aug
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  6. 2024Genie: Generative Interactive Environments23 Feb
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  8. 2023Learning Interactive Real-World Simulators9 Oct
  9. 2022Scalable Diffusion Models with Transformers19 Dec
  10. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  12. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 34

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  9. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  12. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
22 more
  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  2. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  3. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  4. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  5. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  6. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  7. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  8. 2024Navigation World Modelscited by 262
  9. 2024Global Structure-from-Motion Revisitedcited by 215
  10. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  11. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  12. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  13. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  14. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  15. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  16. 2024Video as the New Language for Real-World Decision Makingcited by 86
  17. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 79
  18. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  19. 2021Playable Video Generationcited by 63
  20. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  21. 2024Playable Game Generationcited by 26
  22. 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 57; 2026: 67.

124 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  2. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  3. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  4. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  5. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  6. 2025Video World Models with Long-term Spatial Memorycited by 124
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  8. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  9. 2025Aether: Geometric-Aware Unified World Modelingcited by 94
  10. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 92
  11. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  12. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 86
38 more
  1. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  2. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  3. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 68
  4. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  5. 20253D and 4D World Modeling: A Surveycited by 64
  6. 2025Long-Context State-Space Video World Modelscited by 59
  7. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  8. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  9. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 46
  10. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  11. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  12. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  13. 2025RoboScape: Physics-informed Embodied World Modelcited by 38
  14. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  15. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 36
  16. 2026World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policycited by 36
  17. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  18. 2026MIND: Benchmarking Memory Consistency and Action Control in World Modelscited by 31
  19. 2025Pack and Force Your Memory: Long-form and Consistent Video Generationcited by 26
  20. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  21. 2025A Survey of Interactive Generative Videocited by 23
  22. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  23. 2026WorldCompass: Reinforcement Learning for Long-Horizon World Modelscited by 22
  24. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
  25. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 20
  26. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 19
  27. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  28. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
  29. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 16
  30. 2025Yan: Foundational Interactive Video Generationcited by 15
  31. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  32. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  33. 2025Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generationcited by 15
  34. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 14
  35. 2026WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPGcited by 13
  36. 2025Reinforcement Learning with Inverse Rewards for World Model Post-trainingcited by 12
  37. 2026StableWorld: Towards Stable and Consistent Long Interactive Video Generationcited by 12
  38. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Position: Interactive Generative Video as Next-Generation Game Engine
  2. 2026GameWAM: A World Action Model for Video Games
  3. 2025Matrix-Game: Interactive World Foundation Model
  4. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agents
  5. 2026Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
  6. 2024Moonshine: Distilling Game Content Generators into Steerable Generative Models
  7. 2026DreamX-World 1.0: A General-Purpose Interactive World Model
  8. 2026Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware