# Learning to Decouple the Lights for 3D Face Texture Modeling

arXiv: https://arxiv.org/abs/2412.08524 · HTML: https://mlsanity.com/paper/2412.08524/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 28 of 28)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations](https://mlsanity.com/paper/2203.14510/) (2022)
- [A Morphable Face Albedo Model](https://mlsanity.com/paper/2004.02711/) (2020)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies](https://mlsanity.com/paper/2010.05562/) (2020)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)
- [FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction](https://mlsanity.com/paper/2211.13874/) (2022)
- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [Practical Face Reconstruction via Differentiable Ray Tracing](https://mlsanity.com/paper/2101.05356/) (2021)
- [Face Parsing with RoI Tanh-Warping](https://mlsanity.com/paper/1906.01342/) (2019)
- [Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation](https://mlsanity.com/paper/2106.09614/) (2021)
- [FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio](https://mlsanity.com/paper/2403.01901/) (2024)
- [S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image](https://mlsanity.com/paper/2203.07732/) (2022)
- [Unsupervised Portrait Shadow Removal via Generative Priors](https://mlsanity.com/paper/2108.03466/) (2021)
- [NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images](https://mlsanity.com/paper/2303.14092/) (2023)

## Built on by (most cited first, 1 of 1)

- [WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering](https://mlsanity.com/paper/2512.11237/) (2025)

## Abstract (arXiv)

Existing research has made impressive strides in reconstructing human facial shapes and textures from images with well-illuminated faces and minimal external occlusions. Nevertheless, it remains challenging to recover accurate facial textures from scenarios with complicated illumination affected by external occlusions, e.g. a face that is partially obscured by items such as a hat. Existing works based on the assumption of single and uniform illumination cannot correctly process these data. In this work, we introduce a novel approach to model 3D facial textures under such unnatural illumination. Instead of assuming single illumination, our framework learns to imitate the unnatural illumination as a composition of multiple separate light conditions combined with learned neural representations, named Light Decoupling. According to experiments on both single images and video sequences, we demonstrate the effectiveness of our approach in modeling facial textures under challenging illumination affected by occlusions. Please check https://tianxinhuang.github.io/projects/Deface for our videos and codes.