# Relightable 3D Head Portraits from a Smartphone Video

arXiv: https://arxiv.org/abs/2012.09963 · HTML: https://mlsanity.com/paper/2012.09963/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [Neural Video Portrait Relighting in Real-time via Consistency Modeling](https://mlsanity.com/paper/2104.00484/) (2021)
- [SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage](https://mlsanity.com/paper/2204.03648/) (2022)
- [High-Res Facial Appearance Capture from Polarized Smartphone Images](https://mlsanity.com/paper/2212.01160/) (2022)
- [Learning a 3D Morphable Face Reflectance Model from Low-cost Data](https://mlsanity.com/paper/2303.11686/) (2023)
- [NPBG++: Accelerating Neural Point-Based Graphics](https://mlsanity.com/paper/2203.13318/) (2022)
- [SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis](https://mlsanity.com/paper/2204.10211/) (2022)
- [Personalized Video Relighting With an At-Home Light Stage](https://mlsanity.com/paper/2311.08843/) (2023)
- [Democratizing the Creation of Animatable Facial Avatars](https://mlsanity.com/paper/2401.16534/) (2024)

## Abstract (arXiv)

In this work, a system for creating a relightable 3D portrait of a human head is presented. Our neural pipeline operates on a sequence of frames captured by a smartphone camera with the flash blinking (flash-no flash sequence). A coarse point cloud reconstructed via structure-from-motion software and multi-view denoising is then used as a geometric proxy. Afterwards, a deep rendering network is trained to regress dense albedo, normals, and environmental lighting maps for arbitrary new viewpoints. Effectively, the proxy geometry and the rendering network constitute a relightable 3D portrait model, that can be synthesized from an arbitrary viewpoint and under arbitrary lighting, e.g. directional light, point light, or an environment map. The model is fitted to the sequence of frames with human face-specific priors that enforce the plausibility of albedo-lighting decomposition and operates at the interactive frame rate. We evaluate the performance of the method under varying lighting conditions and at the extrapolated viewpoints and compare with existing relighting methods.