mlsanityData to 14 Oct 2026

A Survey of Interactive Generative Video

arXiv 2504.21853 ↗First posted 30 Apr 2025cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback. We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation. To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence. Furthermore, we systematically analyze the technical challenges and future directions in realizing each component for an ideal IGV system, such as achieving real-time generation, enabling open-domain control, maintaining long-term coherence, simulating accurate physics, and integrating causal reasoning. We believe that this systematic analysis will facilitate future research and development in the field of IGV, ultimately advancing the technology toward more sophisticated and practical applications.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 139 of its 158 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

139 of 158 references.

References · 139

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  8. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  12. 2022Video Diffusion Modelscited by 2,305
38 more
  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  3. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2023Mastering Diverse Domains through World Modelscited by 1,253
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  13. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  14. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  15. 2019An Introduction to Variational Autoencoderscited by 1,000
  16. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  17. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  18. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  19. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  20. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  21. 2022A Generalist Agentcited by 808
  22. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  23. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  24. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  25. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  26. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  27. 2024Genie: Generative Interactive Environmentscited by 747
  28. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  29. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  30. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  31. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  32. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  33. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  34. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  35. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  36. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  37. 2017Generative Adversarial Networks: An Overviewcited by 590
  38. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584

50 most cited of 139 All 139 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  2. 2025Training Agents Inside of Scalable World Modelscited by 136
  3. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  4. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  5. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
  6. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  7. 2025Yan: Foundational Interactive Video Generationcited by 15
  8. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  9. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  10. 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 13
  11. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  12. 2026MemLearner: Learning to Query Context memory for Video World Modelscited by 9
11 more
  1. 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusioncited by 8
  2. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  3. 2026Echo-Memory: A Controlled Study of Memory in Action World Modelscited by 6
  4. 2026Marionette: Predicting World States, Rendering Geometry, Painting Appearancecited by 6
  5. 2025From Masks to Worlds: A Hitchhiker's Guide to World Modelscited by 5
  6. 2026From Pixels to States: Rethinking Interactive World Models as Game Enginescited by 4
  7. 2025SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generationcited by 3
  8. 2025VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesiscited by 3
  9. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  10. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  11. 2026RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstructioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Models
  2. 2025FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework
  3. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models
  4. 2026Autonomous Video Generation with Counterfactual Controllability for Self-Evolving World Models
  5. 2025Controllable Video Generation: A Survey
  6. 2025Pre-Trained Video Generative Models as World Simulators
  7. 2025I2V3D: Controllable image-to-video generation with 3D guidance
  8. 2026When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions