# 3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos

arXiv: https://arxiv.org/abs/2403.01444 · HTML: https://mlsanity.com/paper/2403.01444/

## Key facts

- Cited by 160 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 75 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 30 of 53)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)

## Built on by (most cited first, 30 of 160)

- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos](https://mlsanity.com/paper/2406.18717/) (2024)
- [FlashGS: Efficient 3D Gaussian Splatting for Large-scale and High-resolution Rendering](https://mlsanity.com/paper/2408.07967/) (2024)
- [3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities](https://mlsanity.com/paper/2407.17418/) (2024)
- [Fully Explicit Dynamic Gaussian Splatting](https://mlsanity.com/paper/2410.15629/) (2024)
- [MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting](https://mlsanity.com/paper/2410.07707/) (2024)
- [Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting](https://mlsanity.com/paper/2410.20815/) (2024)
- [GFlow: Recovering 4D World from Monocular Video](https://mlsanity.com/paper/2405.18426/) (2024)
- [EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding](https://mlsanity.com/paper/2412.04380/) (2024)
- [Swift4D:Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene](https://mlsanity.com/paper/2503.12307/) (2025)
- [Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting](https://mlsanity.com/paper/2503.16979/) (2025)
- [Image-GS: Content-Adaptive Image Representation via 2D Gaussians](https://mlsanity.com/paper/2407.01866/) (2024)
- [Splatter a Video: Video Gaussian Representation for Versatile Processing](https://mlsanity.com/paper/2406.13870/) (2024)
- [MoDec-GS: Global-to-Local Motion Decomposition and Temporal Interval Adjustment for Compact Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2501.03714/) (2025)
- [V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic Gaussians](https://mlsanity.com/paper/2409.13648/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video](https://mlsanity.com/paper/2503.18421/) (2025)
- [LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors](https://mlsanity.com/paper/2409.03456/) (2024)
- [QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos](https://mlsanity.com/paper/2412.04469/) (2024)
- [GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views](https://mlsanity.com/paper/2411.11363/) (2024)
- [Segment Any 4D Gaussians](https://mlsanity.com/paper/2407.04504/) (2024)
- [DynOMo: Online Point Tracking by Dynamic Online Monocular Gaussian Reconstruction](https://mlsanity.com/paper/2409.02104/) (2024)
- [Deblur4DGS: 4D Gaussian Splatting from Blurry Monocular Video](https://mlsanity.com/paper/2412.06424/) (2024)
- [StreetSurfGS: Scalable Urban Street Surface Reconstruction with Planar-based Gaussian Splatting](https://mlsanity.com/paper/2410.04354/) (2024)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [Advancing Extended Reality with 3D Gaussian Splatting: Innovations and Prospects](https://mlsanity.com/paper/2412.06257/) (2024)
- [Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt](https://mlsanity.com/paper/2404.11474/) (2024)
- [3DGS-CD: 3D Gaussian Splatting-based Change Detection for Physical Object Rearrangement](https://mlsanity.com/paper/2411.03706/) (2024)
- [Dynamics-Aware Gaussian Splatting Streaming Towards Fast On-the-Fly 4D Reconstruction](https://mlsanity.com/paper/2411.14847/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)

## Abstract (arXiv)

Constructing photo-realistic Free-Viewpoint Videos (FVVs) of dynamic scenes from multi-view videos remains a challenging endeavor. Despite the remarkable advancements achieved by current neural rendering techniques, these methods generally require complete video sequences for offline training and are not capable of real-time rendering. To address these constraints, we introduce 3DGStream, a method designed for efficient FVV streaming of real-world dynamic scenes. Our method achieves fast on-the-fly per-frame reconstruction within 12 seconds and real-time rendering at 200 FPS. Specifically, we utilize 3D Gaussians (3DGs) to represent the scene. Instead of the na"ive approach of directly optimizing 3DGs per-frame, we employ a compact Neural Transformation Cache (NTC) to model the translations and rotations of 3DGs, markedly reducing the training time and storage required for each FVV frame. Furthermore, we propose an adaptive 3DG addition strategy to handle emerging objects in dynamic scenes. Experiments demonstrate that 3DGStream achieves competitive performance in terms of rendering speed, image quality, training time, and model storage when compared with state-of-the-art methods.