mlsanityData to 7 Oct 2026

Diffusion Models Are Real-Time Game Engines

arXiv 2408.14837 ↗First posted 27 Aug 2024cs.LG cs.AI cs.CVcited by 282

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

We present GameNGen, the first game engine powered entirely by a neural model that also enables real-time interaction with a complex environment over long trajectories at high quality. When trained on the classic game DOOM, GameNGen extracts gameplay and uses it to generate a playable environment that can interactively simulate new trajectories. GameNGen runs at 20 frames per second on a single TPU and remains stable over extended multi-minute play sessions. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation, even after 5 minutes of auto-regressive generation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations help ensure stable auto-regressive generation over long trajectories, and decoder fine-tuning improves the fidelity of visual details and text.

Key facts. Cited by 282 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 29 of its 40 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

29 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2023Learning Interactive Real-World Simulators9 Oct
  3. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr

Other references · 26

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
14 more
  1. 2018World Modelscited by 1,444
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  3. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  4. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  7. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  8. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  9. 2024Rolling Diffusion Modelscited by 102
  10. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  11. 2021Playable Video Generationcited by 63
  12. 2018ViZDoom Competitions: Playing Doom from Pixelscited by 50
  13. 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Modelscited by 22
  14. 2018Comparison between CS and JPEG in terms of image compressioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 119; 2026: 144.

282 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  3. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  4. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  5. 2024Latent Action Pretraining from Videoscited by 302
  6. 2024Navigation World Modelscited by 262
  7. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 209
  8. 2025Unified Video Action Modelcited by 195
  9. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  10. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  11. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  12. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
38 more
  1. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  2. 2026Advancing Open-source World Modelscited by 144
  3. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  4. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  5. 2025Video World Models with Long-term Spatial Memorycited by 124
  6. 2025Training Agents Inside of Scalable World Modelscited by 122
  7. 2025Test-Time Training Done Rightcited by 121
  8. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  9. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  10. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  11. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  12. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 93
  13. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 92
  14. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  15. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  16. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  17. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 79
  18. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  19. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 61
  20. 2025Long-Context State-Space Video World Modelscited by 59
  21. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  22. 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
  23. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  24. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  25. 2026Context Forcing: Consistent Autoregressive Video Generation with Long Contextcited by 47
  26. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 46
  27. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  28. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 45
  29. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  30. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  31. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  32. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  33. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  34. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 39
  35. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 38
  36. 2025VRAG: Learning World Models for Interactive Video Generationcited by 37
  37. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  38. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33

50 most cited of 282 All 282 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Multiplayer Interactive World Models with Representation Autoencoders
  2. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
  3. 2023AAMDM: Accelerated Auto-regressive Motion Diffusion Model
  4. 2026Augmentations for Robust and Efficient Imitation Learning in Streamed Video Games
  5. 2026Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware
  6. 2026NitroGen: An Open Foundation Model for Generalist Gaming Agents
  7. 2024Diffusion for World Modeling: Visual Details Matter in Atari
  8. 2026Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory