mlsanityData to 7 Oct 2026

Learning to Simulate Dynamic Environments with GameGAN

arXiv 2005.12126 ↗First posted 25 May 2020cs.CVcited by 86

Not on the map. Why?

Cited by 37 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 16; 2023: 13; 2024: 15; 2025: 14; 2026: 13.

86 papers.

Sort:

By area:

  1. 2024Genie: Generative Interactive Environmentscited by 706
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  4. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  5. 2024Diffusion Models Are Real-Time Game Enginescited by 282
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  7. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  8. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  9. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  10. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  11. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  12. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
38 more
  1. 2022Generating Long Videos of Dynamic Scenescited by 113
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  3. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  4. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  5. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 86
  6. 2024Video as the New Language for Real-World Decision Makingcited by 86
  7. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  8. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  9. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  10. 2021Playable Video Generationcited by 63
  11. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  12. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  13. 2021MarioNette: Self-Supervised Sprite Learningcited by 35
  14. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  15. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  16. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
  17. 2022Weakly-supervised Action Transition Learning for Stochastic Human Motion Predictioncited by 26
  18. 2024Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Videocited by 26
  19. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  20. 2024Game Generation via Large Language Modelscited by 24
  21. 2025A Survey of Interactive Generative Videocited by 23
  22. 2022HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generatorcited by 22
  23. 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Modelscited by 22
  24. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  25. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 18
  26. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 17
  27. 2022A survey on GANs for computer vision: Recent research, analysis and taxonomycited by 15
  28. 2022On the link between conscious function and general intelligence in humans and machinescited by 15
  29. 2023Games for Artificial Intelligence Research: A Review and Perspectivescited by 14
  30. 2023Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-artcited by 14
  31. 2024Unbounded: A Generative Infinite Game of Character Life Simulationcited by 14
  32. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  33. 2026Walk through Paintings: Egocentric World Models from Internet Priorscited by 13
  34. 2022Towards Smooth Video Compositioncited by 12
  35. 2026MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Enginescited by 12
  36. 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 12
  37. 2021Contrastive Learning of Generalized Game Representationscited by 11
  38. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10

50 most cited of 86 All 86 (JSON)