# Representing Long Volumetric Video with Temporal Gaussian Hierarchy

arXiv: https://arxiv.org/abs/2412.09608 · HTML: https://mlsanity.com/paper/2412.09608/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 64 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022)
- [Neural Radiance Flow for 4D View Synthesis and Video Processing](https://mlsanity.com/paper/2012.09790/) (2020)
- [Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle](https://mlsanity.com/paper/2312.03431/) (2023)
- [SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views](https://mlsanity.com/paper/2206.05737/) (2022)
- [Efficient Neural Radiance Fields for Interactive Free-viewpoint Video](https://mlsanity.com/paper/2112.01517/) (2021)
- [Variable Bitrate Neural Fields](https://mlsanity.com/paper/2206.07707/) (2022)
- [3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis](https://mlsanity.com/paper/2404.06270/) (2024)
- [Compressible-composable NeRF via Rank-residual Decomposition](https://mlsanity.com/paper/2205.14870/) (2022)
- [4K4D: Real-Time 4D View Synthesis at 4K Resolution](https://mlsanity.com/paper/2310.11448/) (2023)

## Built on by (most cited first, 30 of 58)

- [ReCamMaster: Camera-Controlled Generative Rendering from A Single Video](https://mlsanity.com/paper/2503.11647/) (2025)
- [FreeTimeGS: Free Gaussian Primitives at Anytime and Anywhere for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2506.05348/) (2025)
- [4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos](https://mlsanity.com/paper/2506.08015/) (2025)
- [Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models](https://mlsanity.com/paper/2507.13344/) (2025)
- [GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects](https://mlsanity.com/paper/2508.14891/) (2025)
- [Multi-View 3D Point Tracking](https://mlsanity.com/paper/2508.21060/) (2025)
- [Faster-GS: Analyzing and Improving Gaussian Splatting Optimization](https://mlsanity.com/paper/2602.09999/) (2026)
- [4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives](https://mlsanity.com/paper/2412.20720/) (2024)
- [TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens](https://mlsanity.com/paper/2604.15239/) (2026)
- [Hybrid 3D-4D Gaussian Splatting for Fast Dynamic Scene Representation](https://mlsanity.com/paper/2505.13215/) (2025)
- [SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars](https://mlsanity.com/paper/2412.15171/) (2024)
- [WideRange4D: Enabling High-Quality 4D Reconstruction with Wide-Range Movements and Scenes](https://mlsanity.com/paper/2503.13435/) (2025)
- [Instant4D: 4D Gaussian Splatting in Minutes](https://mlsanity.com/paper/2510.01119/) (2025)
- [Dynamic Scene Reconstruction: Recent Advance in Real-time Rendering and Streaming](https://mlsanity.com/paper/2503.08166/) (2025)
- [D-FCGS: Feedforward Compression of Dynamic Gaussian Splatting for Free-Viewpoint Videos](https://mlsanity.com/paper/2507.05859/) (2025)
- [4DSloMo: 4D Reconstruction for High Speed Scene with Asynchronous Capture](https://mlsanity.com/paper/2507.05163/) (2025)
- [Prior-Enhanced Gaussian Splatting for Dynamic Scene Reconstruction from Casual Video](https://mlsanity.com/paper/2512.11356/) (2025)
- [Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting](https://mlsanity.com/paper/2503.14698/) (2025)
- [Advances in Radiance Field for Dynamic Scene: From Neural Field to Gaussian Field](https://mlsanity.com/paper/2505.10049/) (2025)
- [Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction](https://mlsanity.com/paper/2505.16533/) (2025)
- [Low-Frequency First: Eliminating Floating Artifacts in 3D Gaussian Splatting](https://mlsanity.com/paper/2508.02493/) (2025)
- [MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis](https://mlsanity.com/paper/2510.07190/) (2025)
- [4C4D: 4 Camera 4D Gaussian Splatting](https://mlsanity.com/paper/2604.04063/) (2026)
- [EvolvingGS: High-Fidelity Streamable Volumetric Video via Evolving 3D Gaussian Representation](https://mlsanity.com/paper/2503.05162/) (2025)
- [UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment](https://mlsanity.com/paper/2506.01802/) (2025)
- [MAPo : Motion-Aware Partitioning of Deformable 3D Gaussian Splatting for High-Fidelity Dynamic Scene Reconstruction](https://mlsanity.com/paper/2508.19786/) (2025)
- [TrackerSplat: Exploiting Point Tracking for Fast and Robust Dynamic 3D Gaussians Reconstruction](https://mlsanity.com/paper/2604.02586/) (2026)
- [Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimation](https://mlsanity.com/paper/2507.11540/) (2025)
- [Adaptive 3D Gaussian Splatting Video Streaming](https://mlsanity.com/paper/2507.14432/) (2025)
- [Splat-SAP: Feed-Forward Gaussian Splatting for Human-Centered Scene with Scale-Aware Point Map Reconstruction](https://mlsanity.com/paper/2511.22704/) (2025)

## Abstract (arXiv)

This paper aims to address the challenge of reconstructing long volumetric videos from multi-view RGB videos. Recent dynamic view synthesis methods leverage powerful 4D representations, like feature grids or point cloud sequences, to achieve high-quality rendering results. However, they are typically limited to short (1~2s) video clips and often suffer from large memory footprints when dealing with longer videos. To solve this issue, we propose a novel 4D representation, named Temporal Gaussian Hierarchy, to compactly model long volumetric videos. Our key observation is that there are generally various degrees of temporal redundancy in dynamic scenes, which consist of areas changing at different speeds. Motivated by this, our approach builds a multi-level hierarchy of 4D Gaussian primitives, where each level separately describes scene regions with different degrees of content change, and adaptively shares Gaussian primitives to represent unchanged scene content over different temporal segments, thus effectively reducing the number of Gaussian primitives. In addition, the tree-like structure of the Gaussian hierarchy allows us to efficiently represent the scene at a particular moment with a subset of Gaussian primitives, leading to nearly constant GPU memory usage during the training or rendering regardless of the video length. Extensive experimental results demonstrate the superiority of our method over alternative methods in terms of training cost, rendering speed, and storage usage. To our knowledge, this work is the first approach capable of efficiently handling minutes of volumetric video data while maintaining state-of-the-art rendering quality. Our project page is available at: https://zju3dv.github.io/longvolcap.