# NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

arXiv: https://arxiv.org/abs/2202.12825 · HTML: https://mlsanity.com/paper/2202.12825/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 69 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 30 of 34)

- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [AGORA: Avatars in Geography Optimized for Regression Analysis](https://mlsanity.com/paper/2104.14643/) (2021)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time](https://mlsanity.com/paper/2202.08614/) (2022)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [Generating 3D People in Scenes without People](https://mlsanity.com/paper/1912.02923/) (2019)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [YOLACT++: Better Real-time Instance Segmentation](https://mlsanity.com/paper/1912.06218/) (2019)
- [DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data](https://mlsanity.com/paper/1912.04302/) (2019)
- [VolumeDeform: Real-time Volumetric Non-rigid Reconstruction](https://mlsanity.com/paper/1603.08161/) (2016)
- [Monocular Real-Time Volumetric Performance Capture](https://mlsanity.com/paper/2007.13988/) (2020)
- [Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction](https://mlsanity.com/paper/2012.01451/) (2020)
- [MonoPerfCap: Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1708.02136/) (2017)
- [Dynamic Surface Function Networks for Clothed Human Bodies](https://mlsanity.com/paper/2104.03978/) (2021)
- [NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras](https://mlsanity.com/paper/2103.07700/) (2021)
- [Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions](https://mlsanity.com/paper/2108.00362/) (2021)

## Built on by (most cited first, 30 of 44)

- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos](https://mlsanity.com/paper/2304.04452/) (2023)
- [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips](https://mlsanity.com/paper/2309.05663/) (2023)
- [HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting](https://mlsanity.com/paper/2312.03461/) (2023)
- [NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions](https://mlsanity.com/paper/2212.07626/) (2022)
- [Human Performance Modeling and Rendering via Neural Animated Mesh](https://mlsanity.com/paper/2209.08468/) (2022)
- [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023)
- [Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments](https://mlsanity.com/paper/2301.02667/) (2023)
- [I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions](https://mlsanity.com/paper/2312.08869/) (2023)
- [Object pop-up: Can we infer 3D objects and their poses from human interactions alone?](https://mlsanity.com/paper/2306.00777/) (2023)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [HumanGen: Generating Human Radiance Fields with Explicit Priors](https://mlsanity.com/paper/2212.05321/) (2022)
- [HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video](https://mlsanity.com/paper/2304.12281/) (2023)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos](https://mlsanity.com/paper/2409.08353/) (2024)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Stream](https://mlsanity.com/paper/2304.03184/) (2023)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams](https://mlsanity.com/paper/2312.01407/) (2023)
- [TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions](https://mlsanity.com/paper/2412.06334/) (2024)
- [CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images](https://mlsanity.com/paper/2308.12288/) (2023)
- [SEGA: Drivable 3D Gaussian Head Avatar from a Single Image](https://mlsanity.com/paper/2504.14373/) (2025)
- [Compositional 3D Human-Object Neural Animation](https://mlsanity.com/paper/2304.14070/) (2023)
- [ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusion](https://mlsanity.com/paper/2509.07920/) (2025)
- [RePerformer: Immersive Human-centric Volumetric Videos from Playback to Photoreal Reperformance](https://mlsanity.com/paper/2503.12242/) (2025)
- [FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models](https://mlsanity.com/paper/2306.00783/) (2023)
- [Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures](https://mlsanity.com/paper/2412.13183/) (2024)
- [MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation](https://mlsanity.com/paper/2312.10120/) (2023)
- [BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video](https://mlsanity.com/paper/2502.08297/) (2025)
- [GASPACHO: Gaussian Splatting for Controllable Humans and Objects](https://mlsanity.com/paper/2503.09342/) (2025)

## Abstract (arXiv)

4D modeling of human-object interactions is critical for numerous applications. However, efficient volumetric capture and rendering of complex interaction scenarios, especially from sparse inputs, remain challenging. In this paper, we propose NeuralHOFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. It marries traditional non-rigid fusion with recent neural implicit modeling and blending advances, where the captured humans and objects are layerwise disentangled. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion, as well as a template-aid robust object tracking pipeline. Our scheme enables detailed and complete geometry generation under complex interactions and occlusions. Moreover, we introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains to obtain photo-realistic results. Extensive experiments demonstrate the effectiveness and efficiency of our approach in synthesizing photo-realistic free-view results under complex human-object interactions.