# QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos

arXiv: https://arxiv.org/abs/2412.04469 · HTML: https://mlsanity.com/paper/2412.04469/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 96 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction](https://mlsanity.com/paper/2309.13101/) (2023)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS](https://mlsanity.com/paper/2311.17245/) (2023)
- [Compact 3D Gaussian Representation for Radiance Field](https://mlsanity.com/paper/2311.13681/) (2023)
- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS](https://mlsanity.com/paper/2312.04564/) (2023)
- [3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos](https://mlsanity.com/paper/2403.01444/) (2024)
- [HUGS: Human Gaussian Splats](https://mlsanity.com/paper/2311.17910/) (2023)
- [GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation](https://mlsanity.com/paper/2403.12365/) (2024)
- [4K4D: Real-Time 4D View Synthesis at 4K Resolution](https://mlsanity.com/paper/2310.11448/) (2023)
- [Neural Trajectory Fields for Dynamic Novel View Synthesis](https://mlsanity.com/paper/2105.05994/) (2021)
- [NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing](https://mlsanity.com/paper/2202.06088/) (2022)
- [INV: Towards Streaming Incremental Neural Videos](https://mlsanity.com/paper/2302.01532/) (2023)

## Built on by (most cited first, 25 of 25)

- [ReCon-GS: Continuum-Preserved Gaussian Streaming for Fast and Compact Reconstruction of Dynamic Scenes](https://mlsanity.com/paper/2509.24325/) (2025)
- [GSCodec Studio: A Modular Framework for Gaussian Splat Compression](https://mlsanity.com/paper/2506.01822/) (2025)
- [MASIV: Toward Material-Agnostic System Identification from Videos](https://mlsanity.com/paper/2508.01112/) (2025)
- [D-FCGS: Feedforward Compression of Dynamic Gaussian Splatting for Free-Viewpoint Videos](https://mlsanity.com/paper/2507.05859/) (2025)
- [Optimized Minimal 4D Gaussian Splatting](https://mlsanity.com/paper/2510.03857/) (2025)
- [Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2505.01235/) (2025)
- [4D-MoDe: Towards Editable and Scalable Volumetric Streaming via Motion-Decoupled 4D Gaussian Compression](https://mlsanity.com/paper/2509.17506/) (2025)
- [StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video](https://mlsanity.com/paper/2511.06046/) (2025)
- [Splat-SAP: Feed-Forward Gaussian Splatting for Human-Centered Scene with Scale-Aware Point Map Reconstruction](https://mlsanity.com/paper/2511.22704/) (2025)
- [LoD-Structured 3D Gaussian Splatting for Streaming Video Reconstruction](https://mlsanity.com/paper/2601.18475/) (2026)
- [MoRGS: Efficient Per-Gaussian Motion Reasoning for Streamable Dynamic 3D Scenes](https://mlsanity.com/paper/2603.25042/) (2026)
- [Do Flat Minima Improve Sparse Novel View Synthesis?](https://mlsanity.com/paper/2511.17918/) (2025)
- [Time-Archival Camera Virtualization for Sports and Visual Performances](https://mlsanity.com/paper/2602.15181/) (2026)
- [MGStream: Motion-aware 3D Gaussian for Streamable Dynamic Scene Reconstruction](https://mlsanity.com/paper/2505.13839/) (2025)
- [Fast and Robust Deformable 3D Gaussian Splatting](https://mlsanity.com/paper/2603.20857/) (2026)
- [CAGS: Color-Adaptive Volumetric Video Streaming with Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2605.09279/) (2026)
- [3D Gaussian Splatting for Efficient Retrospective Dynamic Scene Novel View Synthesis with a Standardized Benchmark](https://mlsanity.com/paper/2605.12437/) (2026)
- [CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Delivery](https://mlsanity.com/paper/2605.18054/) (2026)
- [DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming](https://mlsanity.com/paper/2606.28840/) (2026)
- [QuARC-GS: Quantized Anchored Residual Coding for Compact Dynamic Scene Streaming with Gaussian Splatting](https://mlsanity.com/paper/2608.18285/) (2026)
- [S2GS: Structured Sparse Gaussian Streaming for Efficient Free-Viewpoint Video Reconstruction on Edge-IoT Devices](https://mlsanity.com/paper/2608.19639/) (2026)
- [Racing in Volume with Flow Ensembles](https://mlsanity.com/paper/2609.16310/) (2026)
- [DecoGS: Adaptive Static-Dynamic Decoupling of 3D Gaussians for Free-Viewpoint Video Streaming](https://mlsanity.com/paper/2609.17230/) (2026)
- [Robust, Estimator-Agnostic Dynamic 3DGS Compression](https://mlsanity.com/paper/2609.25633/) (2026)
- [Gaussian Splatting-based Volumetric Video Compression with Sparse 4D Anchors](https://mlsanity.com/paper/2609.33969/) (2026)

## Abstract (arXiv)

Online free-viewpoint video (FVV) streaming is a challenging problem, which is relatively under-explored. It requires incremental on-the-fly updates to a volumetric representation, fast training and rendering to satisfy real-time constraints and a small memory footprint for efficient transmission. If achieved, it can enhance user experience by enabling novel applications, e.g., 3D video conferencing and live volumetric video broadcast, among others. In this work, we propose a novel framework for QUantized and Efficient ENcoding (QUEEN) for streaming FVV using 3D Gaussian Splatting (3D-GS). QUEEN directly learns Gaussian attribute residuals between consecutive frames at each time-step without imposing any structural constraints on them, allowing for high quality reconstruction and generalizability. To efficiently store the residuals, we further propose a quantization-sparsity framework, which contains a learned latent-decoder for effectively quantizing attribute residuals other than Gaussian positions and a learned gating module to sparsify position residuals. We propose to use the Gaussian viewspace gradient difference vector as a signal to separate the static and dynamic content of the scene. It acts as a guide for effective sparsity learning and speeds up training. On diverse FVV benchmarks, QUEEN outperforms the state-of-the-art online FVV methods on all metrics. Notably, for several highly dynamic scenes, it reduces the model size to just 0.7 MB per frame while training in under 5 sec and rendering at 350 FPS. Project website is at https://research.nvidia.com/labs/amri/projects/queen