A Survey of Interactive Generative Video
Not on the map. Why?
Abstract (arXiv)
Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback. We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation. To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence. Furthermore, we systematically analyze the technical challenges and future directions in realizing each component for an ideal IGV system, such as achieving real-time generation, enabling open-domain control, maintaining long-term coherence, simulating accurate physics, and integrating causal reasoning. We believe that this systematic analysis will facilitate future research and development in the field of IGV, ultimately advancing the technology toward more sophisticated and practical applications.
Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 139 of its 158 references that are in the map. First posted on arXiv 30 Apr 2025.
Builds on
139 of 158 references.
References · 139
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
- 2022Video Diffusion Modelscited by 2,305
38 more
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
- 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
- 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
- 2023Mastering Diverse Domains through World Modelscited by 1,253
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
- 2019An Introduction to Variational Autoencoderscited by 1,000
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
- 2022A Generalist Agentcited by 808
- 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
- 2024Genie: Generative Interactive Environmentscited by 747
- 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
- 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
- 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
- 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
- 2024Autoregressive Image Generation without Vector Quantizationcited by 651
- 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
- 2017Generative Adversarial Networks: An Overviewcited by 590
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
50 most cited of 139 All 139 (JSON)
Built on by
Citing papers per year: 2025: 12; 2026: 11.
23 papers.
Sort:
By area:
- 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
- 2025Training Agents Inside of Scalable World Modelscited by 136
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
- 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
- 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
- 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
- 2025Yan: Foundational Interactive Video Generationcited by 15
- 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
- 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 13
- 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
- 2026MemLearner: Learning to Query Context memory for Video World Modelscited by 9
11 more
- 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusioncited by 8
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
- 2026Echo-Memory: A Controlled Study of Memory in Action World Modelscited by 6
- 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearancecited by 6
- 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
- 2026From Pixels to States: Rethinking Interactive World Models as Game Enginescited by 4
- 2025SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generationcited by 3
- 2025VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesiscited by 3
- 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
- 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
- 2026RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstructioncited by 1
- 2026RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction28 Aug 2026
- 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearance14 Aug 2026
- 2026From Pixels to States: Rethinking Interactive World Models as Game Engines15 Jul 2026
- 2026MemLearner: Learning to Query Context memory for Video World Models30 Jun 2026
- 2026Echo-Memory: A Controlled Study of Memory in Action World Models8 Jun 2026
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends31 May 2026
- 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models22 May 2026
- 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Models20 Apr 2026
- 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Models6 Apr 2026
- 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videos10 Feb 2026
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos6 Feb 2026
- 2025VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis22 Dec 2025
11 more
- 2025SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation1 Dec 2025
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmap11 Nov 2025
- 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies2 Nov 2025
- 2025A Step Toward World Models: A Survey on Robotic Manipulation31 Oct 2025
- 2025From Masks to Worlds: A Hitchhiker's Guide to World Models23 Oct 2025
- 2025Training Agents Inside of Scalable World Models29 Sep 2025
- 2025Yan: Foundational Interactive Video Generation12 Aug 2025
- 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling10 Jul 2025
- 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion17 Jun 2025
- 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning10 Jun 2025
- 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval3 Jun 2025
- 2025Training Agents Inside of Scalable World Modelscited by 136
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
- 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
- 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
- 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
- 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 13
- 2026MemLearner: Learning to Query Context memory for Video World Modelscited by 9
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
- 2026Echo-Memory: A Controlled Study of Memory in Action World Modelscited by 6
- 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearancecited by 6
- 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
12 most cited of 15 All 15 (JSON)
- 2026RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction28 Aug 2026
- 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearance14 Aug 2026
- 2026From Pixels to States: Rethinking Interactive World Models as Game Engines15 Jul 2026
- 2026MemLearner: Learning to Query Context memory for Video World Models30 Jun 2026
- 2026Echo-Memory: A Controlled Study of Memory in Action World Models8 Jun 2026
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends31 May 2026
- 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models22 May 2026
- 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Models20 Apr 2026
- 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Models6 Apr 2026
- 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos6 Feb 2026
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmap11 Nov 2025
- 2025A Step Toward World Models: A Survey on Robotic Manipulation31 Oct 2025
12 newest of 15 All 15 (JSON)
- 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
- 2025Yan: Foundational Interactive Video Generationcited by 15
- 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
- 2025SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generationcited by 3
- 2025VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesiscited by 3
- 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videos10 Feb 2026
- 2025VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis22 Dec 2025
- 2025SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation1 Dec 2025
- 2025Yan: Foundational Interactive Video Generation12 Aug 2025
- 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval3 Jun 2025
Similar, not cited
Close in wording, no citation either way
- 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Models
- 2025FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework
- 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models
- 2026Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models
- 2025Controllable Video Generation: A Survey
- 2025Pre-Trained Video Generative Models as World Simulators
- 2025I2V3D: Controllable image-to-video generation with 3D guidance
- 2026When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions