# Volumetric Capture of Humans with a Single RGBD Camera via Semi-Parametric Learning

arXiv: https://arxiv.org/abs/1905.12162 · HTML: https://mlsanity.com/paper/1905.12162/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 13 of 13)

- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Neural Re-Rendering of Humans from a Single Image](https://mlsanity.com/paper/2101.04104/) (2021)
- [NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras](https://mlsanity.com/paper/2103.07700/) (2021)
- [RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream](https://mlsanity.com/paper/2104.14837/) (2021)
- [End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent](https://mlsanity.com/paper/2204.05513/) (2022)
- [Free-Viewpoint RGB-D Human Performance Capture and Rendering](https://mlsanity.com/paper/2112.13889/) (2021)
- [Spatiotemporal Texture Reconstruction for Dynamic Objects Using a Single RGB-D Camera](https://mlsanity.com/paper/2108.09007/) (2021)
- [You Only Train Once: Multi-Identity Free-Viewpoint Neural Human Rendering from Monocular Videos](https://mlsanity.com/paper/2303.05835/) (2023)
- [See360: Novel Panoramic View Interpolation](https://mlsanity.com/paper/2401.03431/) (2024)
- [Uni-Render: A Unified Accelerator for Real-Time Rendering Across Diverse Neural Renderers](https://mlsanity.com/paper/2503.23644/) (2025)
- [Joint 3D Human Shape Recovery and Pose Estimation from a Single Image with Bilayer Graph](https://mlsanity.com/paper/2110.08472/) (2021)
- [LookinGoodπ: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture](https://mlsanity.com/paper/2112.08037/) (2021)

## Abstract (arXiv)

Volumetric (4D) performance capture is fundamental for AR/VR content generation. Whereas previous work in 4D performance capture has shown impressive results in studio settings, the technology is still far from being accessible to a typical consumer who, at best, might own a single RGBD sensor. Thus, in this work, we propose a method to synthesize free viewpoint renderings using a single RGBD camera. The key insight is to leverage previously seen "calibration" images of a given user to extrapolate what should be rendered in a novel viewpoint from the data available in the sensor. Given these past observations from multiple viewpoints, and the current RGBD image from a fixed view, we propose an end-to-end framework that fuses both these data sources to generate novel renderings of the performer. We demonstrate that the method can produce high fidelity images, and handle extreme changes in subject pose and camera viewpoints. We also show that the system generalizes to performers not seen in the training data. We run exhaustive experiments demonstrating the effectiveness of the proposed semi-parametric model (i.e. calibration images available to the neural network) compared to other state of the art machine learned solutions. Further, we compare the method with more traditional pipelines that employ multi-view capture. We show that our framework is able to achieve compelling results, with substantially less infrastructure than previously required.