mlsanityData to 14 Oct 2026

Genie: Generative Interactive Environments

arXiv 2402.15391 ↗First posted 23 Feb 2024cs.LG cs.AI cs.CVcited by 747

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

Key facts. Cited by 747 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 67 of its 80 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

67 of 80 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2023Learning Interactive Real-World Simulators9 Oct
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  4. 2022MAGVIT: Masked Generative Video Transformer10 Dec
  5. 2022MaskGIT: Masked Generative Image Transformer8 Feb
  6. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale22 Oct

Other references · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022Video Diffusion Modelscited by 2,305
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  5. 2020Mastering Atari with Discrete World Modelscited by 1,007
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  7. 2022A Generalist Agentcited by 808
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  11. 2018Behavioral Cloning from Observationcited by 492
  12. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  13. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  14. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  15. 2022Transformers are Sample-Efficient World Modelscited by 328
  16. 2020Query-Key Normalization for Transformerscited by 280
  17. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  18. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  19. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  20. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  21. 2020Spatial-Temporal Transformer Networks for Traffic Flow Forecastingcited by 180
  22. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  23. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  24. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  25. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 137
  26. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  27. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  28. 2023Learning to Act without Actionscited by 121
  29. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  30. 2016Video Pixel Networkscited by 108
  31. 2018Imitating Latent Policies from Observationcited by 101
  32. 2019AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligencecited by 86
  33. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  34. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  35. 2017Recurrent Environment Simulatorscited by 68
  36. 2023Level Generation Through Large Language Modelscited by 68
  37. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
  38. 2021Playable Video Generationcited by 64

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 77; 2025: 228; 2026: 442.

747 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  3. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  4. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  5. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  6. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  7. 2024Latent Action Pretraining from Videoscited by 321
  8. 2026Causal World Modeling for Robot Controlcited by 321
  9. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  10. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  11. 2024Diffusion Policy Policy Optimizationcited by 280
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
38 more
  1. 2024Navigation World Modelscited by 272
  2. 2025Motus: A Unified Latent Action World Modelcited by 251
  3. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  4. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  5. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  6. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
  7. 2025Unified Video Action Modelcited by 205
  8. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  9. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 189
  10. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 186
  11. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  12. 2026Advancing Open-source World Modelscited by 154
  13. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 149
  14. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
  15. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 137
  16. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  17. 2025Unified Vision-Language-Action Modelcited by 134
  18. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  19. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  20. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 124
  21. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
  22. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  23. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  24. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 109
  25. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 106
  26. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 100
  27. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 100
  28. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 97
  29. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  30. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  31. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  32. 2024Video as the New Language for Real-World Decision Makingcited by 87
  33. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  34. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  35. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  36. 2024World Models for Autonomous Driving: An Initial Surveycited by 80
  37. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 80
  38. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Controlcited by 80

50 most cited of 747 All 747 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
  2. 2026OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
  3. 2025Dreamland: Controllable World Creation with Simulator and Generative Models
  4. 2026MiniWorld: Democratizing the Training of Video World Models from Scratch
  5. 2026V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
  6. 2026StateCraft: Augmenting Video World Models with Abstract-State Reasoning
  7. 2026Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation
  8. 2025World-in-World: World Models in a Closed-Loop World