mlsanityData to 14 Oct 2026

Playable Video Generation

arXiv 2101.12195 ↗First posted 28 Jan 2021cs.CV cs.AIcited by 64

Not on the map. Why?

Cited by 21 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces the unsupervised learning problem of playable video generation (PVG). In PVG, we aim at allowing a user to control the generated video by selecting a discrete action at every time step as when playing a video game. The difficulty of the task lies both in learning semantically consistent actions and in generating realistic videos conditioned on the user input. We propose a novel framework for PVG that is trained in a self-supervised manner on a large dataset of unlabelled videos. We employ an encoder-decoder architecture where the predicted action labels act as bottleneck. The network is constrained to learn a rich action space using, as main driving loss, a reconstruction loss on the generated video. We demonstrate the effectiveness of the proposed approach on several datasets with wide environment variety. Further details, code and examples are available on our project page willi-menapace.github.io/playable-video-generation-website.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 28 Jan 2021.

Builds on

39 of 50 references.

References · 39

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  5. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  7. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  8. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2020First Order Motion Model for Image Animationcited by 704
  11. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  12. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 455
27 more
  1. 2018Video-to-Video Synthesiscited by 390
  2. 2018Everybody Dance Nowcited by 315
  3. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
  4. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  5. 2017Stochastic Variational Video Predictioncited by 234
  6. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
  7. 2019Scaling Autoregressive Video Modelscited by 190
  8. 2018Stochastic Adversarial Video Predictioncited by 173
  9. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  10. 2019Temporal Cycle-Consistency Learningcited by 154
  11. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  12. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  13. 2018Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shiftcited by 109
  14. 2020Stochastic Latent Residual Video Predictioncited by 108
  15. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  16. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  17. 2017Recurrent Environment Simulatorscited by 68
  18. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
  19. 2017Unsupervised Learning of Long-Term Motion Dynamics for Videoscited by 37
  20. 2019Unsupervised Keypoint Learning for Guiding Class-Conditional Video Predictioncited by 36
  21. 2019Adversarial Video Generation on Complex Datasetscited by 33
  22. 2020Vid2Player: Controllable Video Sprites that Behave and Appear like Professional Tennis Playerscited by 30
  23. 2018Learning what you can do before doing anythingcited by 26
  24. 2019Vid2Game: Controllable Characters Extracted from Real-World Videoscited by 18
  25. 2015Anticipating Visual Representations from Unlabeled Videocited by 14
  26. 2019Action-conditioned Benchmarking of Robotic Video Prediction Models: a Comparative Studycited by 8
  27. 2020Learning to Cluster under Domain Shiftcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 9; 2024: 11; 2025: 17; 2026: 10.

64 papers.

Sort:

By area:

  1. 2024Genie: Generative Interactive Environmentscited by 747
  2. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  3. 2021Motion Representations for Articulated Animationcited by 255
  4. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  5. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  6. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  7. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  8. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  9. 2023Learning to Act without Actionscited by 121
  10. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  11. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  12. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
38 more
  1. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  2. 2025Long-Context State-Space Video World Modelscited by 64
  3. 2021StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGANcited by 47
  4. 2026Learning Latent Action World Models In The Wildcited by 47
  5. 2025CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrationscited by 41
  6. 2025What Do Latent Action Models Actually Learn?cited by 32
  7. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  8. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  9. 2024Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Videocited by 26
  10. 2024Playable Game Generationcited by 26
  11. 2025Generative Physical AI in Vision: A Surveycited by 24
  12. 2025A Survey of Interactive Generative Videocited by 23
  13. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  14. 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Modelscited by 22
  15. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generationcited by 22
  16. 2021Learning Interactive Driving Policies via Data-driven Simulationcited by 17
  17. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  18. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  19. 2021Strumming to the Beat: Audio-Conditioned Contrastive Video Texturescited by 14
  20. 2022Jointly Harnessing Prior Structures and Temporal Consistency for Sign Language Video Generationcited by 14
  21. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 14
  22. 2026Code World Model: Coding Agent as World Braincited by 14
  23. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  24. 2024Move Anything with Layered Scene Diffusioncited by 12
  25. 2023A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decisioncited by 11
  26. 2026MASS: Multiplayer World Models with Authoritative Shared Statecited by 11
  27. 2025Exploration-Driven Generative Interactive Environmentscited by 9
  28. 2022Video2StyleGAN: Disentangling Local and Global Variations in a Videocited by 8
  29. 2022SSGVS: Semantic Scene Graph-to-Video Synthesiscited by 8
  30. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  31. 2021Layered Controllable Video Generationcited by 7
  32. 2023Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generationcited by 7
  33. 2022Controllable Video Generation through Global and Local Motion Dynamicscited by 6
  34. 2025Model as a Game: On Numerical and Spatial Consistency for Generative Gamescited by 5
  35. 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
  36. 2026ActionParty: Multi-Subject Action Binding in Generative Video Gamescited by 5
  37. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  38. 2024Learning Generative Interactive Environments By Trained Agent Explorationcited by 4

50 most cited of 64 All 64 (JSON)