# NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

arXiv: https://arxiv.org/abs/2202.06088 · HTML: https://mlsanity.com/paper/2202.06088/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 12 Feb 2022.

## Builds on (most cited first, 29 of 29)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [AutoInt: Automatic Integration for Fast Neural Volume Rendering](https://mlsanity.com/paper/2012.01714/) (2020)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [Learning the Depths of Moving People by Watching Frozen People](https://mlsanity.com/paper/1904.11111/) (2019)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data](https://mlsanity.com/paper/1912.04302/) (2019)
- [Layered Neural Rendering for Retiming People in Video](https://mlsanity.com/paper/2009.07833/) (2020)
- [NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras](https://mlsanity.com/paper/2103.07700/) (2021)
- [ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References](https://mlsanity.com/paper/2103.06747/) (2021)
- [iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering](https://mlsanity.com/paper/2108.05577/) (2021)

## Built on by (most cited first, 12 of 12)

- [HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion](https://mlsanity.com/paper/2305.06356/) (2023)
- [Human Performance Modeling and Rendering via Neural Animated Mesh](https://mlsanity.com/paper/2209.08468/) (2022)
- [AvatarGen: A 3D Generative Model for Animatable Human Avatars](https://mlsanity.com/paper/2211.14589/) (2022)
- [Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos](https://mlsanity.com/paper/2409.08353/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos](https://mlsanity.com/paper/2412.04469/) (2024)
- [Dynamic PlenOctree for Adaptive Sampling Refinement in Explicit NeRF](https://mlsanity.com/paper/2307.15333/) (2023)
- [3D Video Loops from Asynchronous Input](https://mlsanity.com/paper/2303.05312/) (2023)
- [Fast Non-Rigid Radiance Fields from Monocularized Data](https://mlsanity.com/paper/2212.01368/) (2022)
- [Splatwizard: A Benchmark Toolkit for 3D Gaussian Splatting Compression](https://mlsanity.com/paper/2512.24742/) (2025)
- [AvatarGen: a 3D Generative Model for Animatable Human Avatars](https://mlsanity.com/paper/2208.00561/) (2022)
- [UVA: Towards Unified Volumetric Avatar for View Synthesis, Pose rendering, Geometry and Texture Editing](https://mlsanity.com/paper/2304.06969/) (2023)

## Abstract (arXiv)

Some of the most exciting experiences that Metaverse promises to offer, for instance, live interactions with virtual characters in virtual environments, require real-time photo-realistic rendering. 3D reconstruction approaches to rendering, active or passive, still require extensive cleanup work to fix the meshes or point clouds. In this paper, we present a neural volumography technique called neural volumetric video or NeuVV to support immersive, interactive, and spatial-temporal rendering of volumetric video contents with photo-realism and in real-time. The core of NeuVV is to efficiently encode a dynamic neural radiance field (NeRF) into renderable and editable primitives. We introduce two types of factorization schemes: a hyper-spherical harmonics (HH) decomposition for modeling smooth color variations over space and time and a learnable basis representation for modeling abrupt density and color changes caused by motion. NeuVV factorization can be integrated into a Video Octree (VOctree) analogous to PlenOctree to significantly accelerate training while reducing memory overhead. Real-time NeuVV rendering further enables a class of immersive content editing tools. Specifically, NeuVV treats each VOctree as a primitive and implements volume-based depth ordering and alpha blending to realize spatial-temporal compositions for content re-purposing. For example, we demonstrate positioning varied manifestations of the same performance at different 3D locations with different timing, adjusting color/texture of the performer's clothing, casting spotlight shadows and synthesizing distance falloff lighting, etc, all at an interactive speed. We further develop a hybrid neural-rasterization rendering framework to support consumer-level VR headsets so that the aforementioned volumetric video viewing and editing, for the first time, can be conducted immersively in virtual 3D space.