# Rate-aware Compression for NeRF-based Volumetric Video

arXiv: https://arxiv.org/abs/2411.05322 · HTML: https://mlsanity.com/paper/2411.05322/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [HexPlane: A Fast Representation for Dynamic Scenes](https://mlsanity.com/paper/2301.09632/) (2023)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Fast Dynamic Radiance Fields with Time-Aware Neural Voxels](https://mlsanity.com/paper/2205.15285/) (2022)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Neural Radiance Flow for 4D View Synthesis and Video Processing](https://mlsanity.com/paper/2012.09790/) (2020)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time](https://mlsanity.com/paper/2202.08614/) (2022)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion](https://mlsanity.com/paper/2305.06356/) (2023)
- [Variable Bitrate Neural Fields](https://mlsanity.com/paper/2206.07707/) (2022)
- [DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2205.15723/) (2022)
- [Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos](https://mlsanity.com/paper/2304.04452/) (2023)
- [VoxelContext-Net: An Octree based Framework for Point Cloud Compression](https://mlsanity.com/paper/2105.02158/) (2021)
- [C3: High-performance and low-complexity neural compression from a single image or video](https://mlsanity.com/paper/2312.02753/) (2023)
- [VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams](https://mlsanity.com/paper/2312.01407/) (2023)
- [TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video](https://mlsanity.com/paper/2312.06713/) (2023)
- [JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression](https://mlsanity.com/paper/2405.14452/) (2024)
- [aiWave: Volumetric Image Compression with 3-D Trained Affine Wavelet-like Transform](https://mlsanity.com/paper/2203.05822/) (2022)
- [ECRF: Entropy-Constrained Neural Radiance Fields Compression with Frequency Domain Optimization](https://mlsanity.com/paper/2311.14208/) (2023)
- [CAwa-NeRF: Instant Learning of Compression-Aware NeRF Features](https://mlsanity.com/paper/2310.14695/) (2023)

## Built on by (most cited first, 8 of 8)

- [Dynamic Scene Reconstruction: Recent Advance in Real-time Rendering and Streaming](https://mlsanity.com/paper/2503.08166/) (2025)
- [Mesh Compression with Quantized Neural Displacement Fields](https://mlsanity.com/paper/2504.01027/) (2025)
- [StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video](https://mlsanity.com/paper/2511.06046/) (2025)
- [Squeeze3D: Extreme Neural Compression with Latent Space Bridging](https://mlsanity.com/paper/2506.07932/) (2025)
- [CATRF: Codec-Adaptive TriPlane Radiance Fields for Volumetric Content Delivery](https://mlsanity.com/paper/2605.18054/) (2026)
- [DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming](https://mlsanity.com/paper/2606.28840/) (2026)
- [Compressing 3D Gaussian Splatting via Cross-Representation Priors](https://mlsanity.com/paper/2609.23005/) (2026)
- [Gaussian Splatting-based Volumetric Video Compression with Sparse 4D Anchors](https://mlsanity.com/paper/2609.33969/) (2026)

## Abstract (arXiv)

The neural radiance fields (NeRF) have advanced the development of 3D volumetric video technology, but the large data volumes they involve pose significant challenges for storage and transmission. To address these problems, the existing solutions typically compress these NeRF representations after the training stage, leading to a separation between representation training and compression. In this paper, we try to directly learn a compact NeRF representation for volumetric video in the training stage based on the proposed rate-aware compression framework. Specifically, for volumetric video, we use a simple yet effective modeling strategy to reduce temporal redundancy for the NeRF representation. Then, during the training phase, an implicit entropy model is utilized to estimate the bitrate of the NeRF representation. This entropy model is then encoded into the bitstream to assist in the decoding of the NeRF representation. This approach enables precise bitrate estimation, thereby leading to a compact NeRF representation. Furthermore, we propose an adaptive quantization strategy and learn the optimal quantization step for the NeRF representations. Finally, the NeRF representation can be optimized by using the rate-distortion trade-off. Our proposed compression framework can be used for different representations and experimental results demonstrate that our approach significantly reduces the storage size with marginal distortion and achieves state-of-the-art rate-distortion performance for volumetric video on the HumanRF and ReRF datasets. Compared to the previous state-of-the-art method TeTriRF, we achieved an approximately -80% BD-rate on the HumanRF dataset and -60% BD-rate on the ReRF dataset.