mlsanityData to 14 Oct 2026

DyST: Towards Dynamic Neural Scene Representations on Real-World Videos

arXiv 2310.06020 ↗First posted 9 Oct 2023cs.CV cs.AI cs.GRcited by 11

Not on the map. Why?

Abstract (arXiv)

Visual understanding of the world goes beyond the semantics and flat structure of individual images. In this work, we aim to capture both the 3D structure and dynamics of real-world scenes from monocular real-world videos. Our Dynamic Scene Transformer (DyST) model leverages recent work in neural scene representation to learn a latent decomposition of monocular real-world videos into scene content, per-view scene dynamics, and camera pose. This separation is achieved through a novel co-training scheme on monocular videos and our new synthetic dataset DySO. DyST learns tangible latent representations for dynamic scenes that enable view generation with separate control over the camera and the content of the scene.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

27 of 31 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  4. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  7. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  8. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  9. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  10. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  11. 2022Kubric: A scalable dataset generatorcited by 381
  12. 2021Advances in Neural Renderingcited by 365
15 more
  1. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  2. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  3. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 227
  4. 2023Robust Dynamic Radiance Fieldscited by 194
  5. 2021H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motioncited by 157
  6. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  7. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 152
  8. 2021NeRF-VAE: A Geometry Aware 3D Scene Generative Modelcited by 112
  9. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  10. 2022Object Scene Representation Transformercited by 90
  11. 2021T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesiscited by 88
  12. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  13. 2022RUST: Latent Neural Scene Representations from Unposed Imagerycited by 43
  14. 2023DORSal: Diffusion for Object-centric Representations of Scenes et alcited by 16
  15. 2022MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Posecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 2.

11 papers.

  1. 2024L4GM: Large 4D Gaussian Reconstruction Modelcited by 144
  2. 2024Controlling Space and Time with Diffusion Modelscited by 66
  3. 2024G3R: Gradient Guided Generalizable Reconstructioncited by 31
  4. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  5. 20254D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Timecited by 14
  6. 2024Moving Off-the-Grid: Scene-Grounded Video Representationscited by 11
  7. 2025Splat4D: Diffusion-Enhanced 4D Gaussian Splatting for Temporally and Spatially Consistent Content Creationcited by 9
  8. 2024Parallelized Spatiotemporal Bindingcited by 2
  9. 2025Neural USD: An object-centric framework for iterative editing and controlcited by 2
  10. 2026GRVS: a Generalizable and Recurrent Approach to Monocular Dynamic View Synthesiscited by 0
  11. 2026RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Videocited by 0