# Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures

arXiv: https://arxiv.org/abs/2412.13183 · HTML: https://mlsanity.com/paper/2412.13183/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 97 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 30 of 52)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects](https://mlsanity.com/paper/2004.02806/) (2020)
- [NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction](https://mlsanity.com/paper/2212.05231/) (2022)
- [Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors](https://mlsanity.com/paper/2105.01859/) (2021)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis](https://mlsanity.com/paper/2312.02155/) (2023)
- [DeepCap: Monocular Human Performance Capture Using Weak Supervision](https://mlsanity.com/paper/2003.08325/) (2020)
- [3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos](https://mlsanity.com/paper/2403.01444/) (2024)
- [Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering](https://mlsanity.com/paper/2109.07448/) (2021)
- [Efficient Neural Radiance Fields for Interactive Free-viewpoint Video](https://mlsanity.com/paper/2112.01517/) (2021)
- [ARAH: Animatable Volume Rendering of Articulated Human SDFs](https://mlsanity.com/paper/2210.10036/) (2022)
- [Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition](https://mlsanity.com/paper/2302.11566/) (2023)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control](https://mlsanity.com/paper/2106.02019/) (2021)
- [TeCH: Text-guided Reconstruction of Lifelike Clothed Humans](https://mlsanity.com/paper/2308.08545/) (2023)
- [ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering](https://mlsanity.com/paper/2312.05941/) (2023)
- [Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing](https://mlsanity.com/paper/2108.08807/) (2021)
- [DoubleFusion: Real-time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor](https://mlsanity.com/paper/1804.06023/) (2018)
- [LiveCap: Real-time Human Performance Capture from Monocular Video](https://mlsanity.com/paper/1810.02648/) (2018)
- [Capturing and Animation of Body and Clothing from Monocular Video](https://mlsanity.com/paper/2210.01868/) (2022)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)

## Built on by (most cited first, 7 of 7)

- [BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video](https://mlsanity.com/paper/2502.08297/) (2025)
- [UMA: Ultra-detailed Human Avatars via Multi-level Surface Alignment](https://mlsanity.com/paper/2506.01802/) (2025)
- [Relightable Holoported Characters: Capturing and Relighting Dynamic Human Performance from Sparse Views](https://mlsanity.com/paper/2512.00255/) (2025)
- [Hyper Diffusion Avatars: Dynamic Human Avatar Generation using Network Weight Space Diffusion](https://mlsanity.com/paper/2509.04145/) (2025)
- [Revisiting Avatar-As-Image: High-Fidelity Registration is All You Need](https://mlsanity.com/paper/2609.11722/) (2026)
- [Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splatting](https://mlsanity.com/paper/2503.09640/) (2025)
- [MUA: Mobile Ultra-detailed Animatable Avatars](https://mlsanity.com/paper/2604.18583/) (2026)

## Abstract (arXiv)

Real-time free-view human rendering from sparse-view RGB inputs is a challenging task due to the sensor scarcity and the tight time budget. To ensure efficiency, recent methods leverage 2D CNNs operating in texture space to learn rendering primitives. However, they either jointly learn geometry and appearance, or completely ignore sparse image information for geometry estimation, significantly harming visual quality and robustness to unseen body poses. To address these issues, we present Double Unprojected Textures, which at the core disentangles coarse geometric deformation estimation from appearance synthesis, enabling robust and photorealistic 4K rendering in real-time. Specifically, we first introduce a novel image-conditioned template deformation network, which estimates the coarse deformation of the human template from a first unprojected texture. This updated geometry is then used to apply a second and more accurate texture unprojection. The resulting texture map has fewer artifacts and better alignment with input views, which benefits our learning of finer-level geometry and appearance represented by Gaussian splats. We validate the effectiveness and efficiency of the proposed method in quantitative and qualitative experiments, which significantly surpasses other state-of-the-art methods. Project page: https://vcai.mpi-inf.mpg.de/projects/DUT/