# VRVVC: Variable-Rate NeRF-Based Volumetric Video Compression

arXiv: https://arxiv.org/abs/2412.11362 · HTML: https://mlsanity.com/paper/2412.11362/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 25 of 25)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [Neural Radiance Flow for 4D View Synthesis and Video Processing](https://mlsanity.com/paper/2012.09790/) (2020)
- [MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes](https://mlsanity.com/paper/2302.12249/) (2023)
- [Tensor4D : Efficient Neural 4D Decomposition for High-fidelity Dynamic Reconstruction and Rendering](https://mlsanity.com/paper/2211.11610/) (2022)
- [HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion](https://mlsanity.com/paper/2305.06356/) (2023)
- [Streaming Radiance Fields for 3D Video Synthesis](https://mlsanity.com/paper/2210.14831/) (2022)
- [Variable Rate Deep Image Compression With a Conditional Autoencoder](https://mlsanity.com/paper/1909.04802/) (2019)
- [DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering](https://mlsanity.com/paper/2307.10173/) (2023)
- [Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation](https://mlsanity.com/paper/2003.02012/) (2020)
- [Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos](https://mlsanity.com/paper/2304.04452/) (2023)
- [Representing Volumetric Videos as Dynamic MLP Maps](https://mlsanity.com/paper/2304.06717/) (2023)
- [Variable Rate Deep Image Compression with Modulated Autoencoder](https://mlsanity.com/paper/1912.05526/) (2019)
- [VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams](https://mlsanity.com/paper/2312.01407/) (2023)
- [TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video](https://mlsanity.com/paper/2312.06713/) (2023)
- [HPC: Hierarchical Progressive Coding Framework for Volumetric Video](https://mlsanity.com/paper/2407.09026/) (2024)
- [ECRF: Entropy-Constrained Neural Radiance Fields Compression with Frequency Domain Optimization](https://mlsanity.com/paper/2311.14208/) (2023)
- [FPO++: Efficient Encoding and Rendering of Dynamic Neural Radiance Fields by Analyzing and Enhancing Fourier PlenOctrees](https://mlsanity.com/paper/2310.20710/) (2023)

## Built on by (most cited first, 14 of 14)

- [BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video](https://mlsanity.com/paper/2502.08297/) (2025)
- [Semantics versus Identity: A Divide-and-Conquer Approach towards Adjustable Medical Image De-Identification](https://mlsanity.com/paper/2507.21703/) (2025)
- [4D-MoDe: Towards Editable and Scalable Volumetric Streaming via Motion-Decoupled 4D Gaussian Compression](https://mlsanity.com/paper/2509.17506/) (2025)
- [AlignGS: Aligning Geometry and Semantics for Robust Indoor Reconstruction from Sparse Views](https://mlsanity.com/paper/2510.07839/) (2025)
- [StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video](https://mlsanity.com/paper/2511.06046/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [PrismGS: Physically-Grounded Anti-Aliasing for High-Fidelity Large-Scale 3D Gaussian Splatting](https://mlsanity.com/paper/2510.07830/) (2025)
- [GenSplatCodec: Feed-Forward Gaussian Splatting Compression via One-Step Diffusion](https://mlsanity.com/paper/2607.24403/) (2026)
- [MultiEgo: A Multi-View Egocentric Video Dataset for 4D Scene Reconstruction](https://mlsanity.com/paper/2512.11301/) (2025)
- [Lightweight High-Fidelity Low-Bitrate Talking Face Compression for 3D Video Conference](https://mlsanity.com/paper/2601.21269/) (2026)
- [CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Delivery](https://mlsanity.com/paper/2605.18054/) (2026)
- [DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming](https://mlsanity.com/paper/2606.28840/) (2026)
- [Struct-GStream: Towards Efficient Free-Viewpoint Video Streaming at Low-Bitrates with Structured 3D Gaussians](https://mlsanity.com/paper/2608.01053/) (2026)
- [Gaussian Splatting-based Volumetric Video Compression with Sparse 4D Anchors](https://mlsanity.com/paper/2609.33969/) (2026)

## Abstract (arXiv)

Neural Radiance Field (NeRF)-based volumetric video has revolutionized visual media by delivering photorealistic Free-Viewpoint Video (FVV) experiences that provide audiences with unprecedented immersion and interactivity. However, the substantial data volumes pose significant challenges for storage and transmission. Existing solutions typically optimize NeRF representation and compression independently or focus on a single fixed rate-distortion (RD) tradeoff. In this paper, we propose VRVVC, a novel end-to-end joint optimization variable-rate framework for volumetric video compression that achieves variable bitrates using a single model while maintaining superior RD performance. Specifically, VRVVC introduces a compact tri-plane implicit residual representation for inter-frame modeling of long-duration dynamic scenes, effectively reducing temporal redundancy. We further propose a variable-rate residual representation compression scheme that leverages a learnable quantization and a tiny MLP-based entropy model. This approach enables variable bitrates through the utilization of predefined Lagrange multipliers to manage the quantization error of all latent representations. Finally, we present an end-to-end progressive training strategy combined with a multi-rate-distortion loss function to optimize the entire framework. Extensive experiments demonstrate that VRVVC achieves a wide range of variable bitrates within a single model and surpasses the RD performance of existing methods across various datasets.