mlsanityData to 14 Oct 2026

GameGen-X: Interactive Open-world Game Video Generation

arXiv 2411.00769 ↗First posted 1 Nov 2024cs.CV cs.AIcited by 128

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

We introduce GameGen-X, the first diffusion transformer model specifically designed for both generating and interactively controlling open-world game videos. This model facilitates high-quality, open-domain generation by simulating an extensive array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, predicting and altering future content based on the current clip, thus allowing for gameplay simulation. To realize this vision, we first collected and built an Open-World Video Game Dataset from scratch. It is the first and largest dataset for open-world game video generation and control, which comprises over a million diverse gameplay video clips sampling from over 150 games with informative captions from GPT-4o. GameGen-X undergoes a two-stage training process, consisting of foundation model pre-training and instruction tuning. Firstly, the model was pre-trained via text-to-video generation and video continuation, endowing it with the capability for long-sequence, high-quality open-domain game video generation. Further, to achieve interactive controllability, we designed InstructNet to incorporate game-related multi-modal control signal experts. This allows the model to adjust latent representations based on user inputs, unifying character interaction and scene content control for the first time in video generation. During instruction tuning, only the InstructNet is updated while the pre-trained foundation model is frozen, enabling the integration of interactive controllability without loss of diversity and quality of generated video content.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 54 of its 65 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

54 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Diffusion Models Are Real-Time Game Engines27 Aug
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  3. 2024Pandora: Towards General World Model with Natural Language Actions and Video States12 Jun
  4. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond6 May
  5. 2024Genie: Generative Interactive Environments23 Feb
  6. 2023Learning Interactive Real-World Simulators9 Oct
  7. 2022Scalable Diffusion Models with Transformers19 Dec
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  9. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  10. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  11. 2018World Modelscited by 1,513
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
33 more
  1. 2022Diffusion Models in Vision: A Surveycited by 1,015
  2. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  3. 2017Localizing Moments in Video with Natural Languagecited by 663
  4. 2023ModelScope Text-to-Video Technical Reportcited by 658
  5. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  8. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  9. 2023CoTracker: It is Better to Track Togethercited by 522
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  11. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  12. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  13. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  14. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  15. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  16. 2020Query-Key Normalization for Transformerscited by 280
  17. 2022Unifying Flow, Stereo and Depth Estimationcited by 275
  18. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
  19. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  20. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  21. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  22. 2020Latent Video Transformercited by 106
  23. 2023VideoLCM: Video Latent Consistency Modelcited by 92
  24. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  25. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 77
  26. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  27. 2021Playable Video Generationcited by 64
  28. 2024VEnhancer: Generative Space-Time Enhancement for Video Generationcited by 64
  29. 2022Composable Text Controls in Latent Space with ODEscited by 28
  30. 2024DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformerscited by 15
  31. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  32. 2024Read to Play (R2-Play): Decision Transformer with Multimodal Game Instructioncited by 4
  33. 2024MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spacescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 57; 2026: 66.

128 papers.

Sort:

By area:

  1. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  2. 2025Unified Video Action Modelcited by 205
  3. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 186
  4. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 170
  5. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  6. 2025Video World Models with Long-term Spatial Memorycited by 136
  7. 2025Unified Vision-Language-Action Modelcited by 134
  8. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  9. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  10. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  11. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 98
  12. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 96
38 more
  1. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80
  2. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 73
  3. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  4. 2025Long-Context State-Space Video World Modelscited by 64
  5. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 61
  6. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 55
  7. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 53
  8. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 52
  9. 2026Infinite Worlds with Versatile Interactionscited by 48
  10. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  11. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 41
  12. 2025RoboScape: Physics-informed Embodied World Modelcited by 40
  13. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
  14. 2026Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memorycited by 40
  15. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 40
  16. 2025VRAG: Learning World Models for Interactive Video Generationcited by 38
  17. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  18. 2026MIND: Benchmarking Memory Consistency and Action Control in World Modelscited by 35
  19. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  20. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  21. 2025ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichmentcited by 27
  22. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 26
  23. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 25
  24. 20253D Scene Generation: A Surveycited by 24
  25. 2025Scalable Policy Evaluation with Video World Modelscited by 24
  26. 2025A Survey of Interactive Generative Videocited by 23
  27. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  28. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 22
  29. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 21
  30. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Frameworkcited by 20
  31. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 19
  32. 2025RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformerscited by 18
  33. 2025GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photographycited by 18
  34. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 18
  35. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 18
  36. 2026Latent Spatial Memory for Video World Modelscited by 17
  37. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 16
  38. 2026LIVE: Long-horizon Interactive Video World Modelingcited by 16

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Playable Game Generation
  2. 2025Pre-Trained Video Generative Models as World Simulators
  3. 2025Matrix-Game: Interactive World Foundation Model
  4. 2026Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
  5. 2026Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware
  6. 2026Generative World Renderer
  7. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
  8. 2026AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report