mlsanityData to 7 Oct 2026

Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video

arXiv 2404.09833 ↗First posted 15 Apr 2024cs.CV cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Creating high-quality and interactive virtual environments, such as games and simulators, often involves complex and costly manual modeling processes. In this paper, we present Video2Game, a novel approach that automatically converts videos of real-world scenes into realistic and interactive game environments. At the heart of our system are three core components:(i) a neural radiance fields (NeRF) module that effectively captures the geometry and visual appearance of the scene; (ii) a mesh module that distills the knowledge from NeRF for faster rendering; and (iii) a physics module that models the interactions and physical dynamics among the objects. By following the carefully designed pipeline, one can construct an interactable and actionable digital replica of the real world. We benchmark our system on both indoor and large-scale outdoor scenes. We show that we can not only produce highly-realistic renderings in real-time, but also build interactive games on top.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 61 of its 93 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

61 of 93 references.

References · 61

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  8. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  9. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  10. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  11. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  12. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
38 more
  1. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  2. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  3. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  4. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  5. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  6. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  7. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  8. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  9. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  10. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  11. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  12. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  13. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  14. 2021Urban Radiance Fieldscited by 263
  15. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  16. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 220
  17. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  18. 2021HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fieldscited by 210
  19. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  20. 2023Text-To-4D Dynamic Scene Generationcited by 204
  21. 2019Pushing the Boundaries of View Extrapolation with Multiplane Imagescited by 191
  22. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  23. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  24. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  25. 2021NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illuminationcited by 135
  26. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  27. 20223D Common Corruptions and Data Augmentationcited by 120
  28. 2019DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatchcited by 114
  29. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  30. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  31. 2022Learning to Generate Realistic LiDAR Point Cloudscited by 91
  32. 2020Learning to Simulate Dynamic Environments with GameGANcited by 86
  33. 2023VR-NeRF: High-Fidelity Virtualized Walkable Spacescited by 86
  34. 2018Layer-structured 3D Scene Inference via View Synthesiscited by 82
  35. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  36. 2020Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Imagecited by 77
  37. 2023UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generationcited by 76
  38. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 13; 2026: 9.

26 papers.

Sort:

By area:

  1. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  2. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  3. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  4. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 40
  5. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  6. 2025Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigationcited by 30
  7. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  8. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  9. 2025Generative Physical AI in Vision: A Surveycited by 24
  10. 2024From an Image to a Scene: Learning to Imagine the World from a Million 360 Videoscited by 23
  11. 2025AE-NeRF: Augmenting Event-Based Neural Radiance Fields for Non-ideal Conditions and Larger Scenecited by 19
  12. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
14 more
  1. 2024Robot Learning with Super-Linear Scalingcited by 10
  2. 2025SIMPACT: Simulation-Enabled Action Planning using Vision-Language Modelscited by 7
  3. 2025ForestSplats: Deformable transient field for Gaussian Splatting in the Wildcited by 6
  4. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
  5. 2025PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animationcited by 4
  6. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  7. 2026FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minutecited by 3
  8. 2025Veta-GS: View-dependent deformable 3D Gaussian Splatting for thermal infrared Novel-view Synthesiscited by 1
  9. 2026Streaming Video Generation with Streaming Force Controlcited by 1
  10. 2026Adaptive Volumetric Mechanical Property Fields Invariant to Resolutioncited by 1
  11. 2025Advancing Text-to-3D Generation with Linearized Lookahead Variational Score Distillationcited by 0
  12. 2026Shell-Supervised Gaussian Splatting for Urban Real-to-Sim Reconstructioncited by 0
  13. 2026BinauralVAE: Spatial Audio Reconstruction For World Modelscited by 0
  14. 2026LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstructioncited by 0