# Photorealistic Facial Texture Inference Using Deep Neural Networks

arXiv: https://arxiv.org/abs/1612.00523 · HTML: https://mlsanity.com/paper/1612.00523/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2016.

## Built on by (most cited first, 30 of 32)

- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [NeuTex: Neural Texture Mapping for Volumetric Neural Rendering](https://mlsanity.com/paper/2103.00762/) (2021)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction](https://mlsanity.com/paper/2211.13874/) (2022)
- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [Practical Face Reconstruction via Differentiable Ray Tracing](https://mlsanity.com/paper/2101.05356/) (2021)
- [Relightify: Relightable 3D Faces from a Single Image via Diffusion Models](https://mlsanity.com/paper/2305.06077/) (2023)
- [Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/2105.07474/) (2021)
- [Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguation](https://mlsanity.com/paper/2205.03962/) (2022)
- [SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image](https://mlsanity.com/paper/2309.01855/) (2023)
- [Facial Geometric Detail Recovery via Implicit Representation](https://mlsanity.com/paper/2203.09692/) (2022)
- [MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation](https://mlsanity.com/paper/2111.01048/) (2021)
- [High-Res Facial Appearance Capture from Polarized Smartphone Images](https://mlsanity.com/paper/2212.01160/) (2022)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [3D-FM GAN: Towards 3D-Controllable Face Manipulation](https://mlsanity.com/paper/2208.11257/) (2022)
- [Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization](https://mlsanity.com/paper/2305.03043/) (2023)
- [Deep Hybrid Self-Prior for Full 3D Mesh Generation](https://mlsanity.com/paper/2108.08017/) (2021)
- [BareSkinNet: De-makeup and De-lighting via 3D Face Reconstruction](https://mlsanity.com/paper/2209.09029/) (2022)
- [Where Do Deep Fakes Look? Synthetic Face Detection via Gaze Tracking](https://mlsanity.com/paper/2101.01165/) (2021)
- [Makeup Extraction of 3D Representation via Illumination-Aware Image Decomposition](https://mlsanity.com/paper/2302.13279/) (2023)
- [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [DreamSalon: A Staged Diffusion Framework for Preserving Identity-Context in Editable Face Generation](https://mlsanity.com/paper/2403.19235/) (2024)
- [Normal and Visibility Estimation of Human Face from a Single Image](https://mlsanity.com/paper/2203.04647/) (2022)
- [Initialization and Alignment for Adversarial Texture Optimization](https://mlsanity.com/paper/2207.14289/) (2022)
- [FSID: Fully Synthetic Image Denoising via Procedural Scene Generation](https://mlsanity.com/paper/2212.03961/) (2022)
- [Face Inverse Rendering via Hierarchical Decoupling](https://mlsanity.com/paper/2301.06733/) (2023)
- [MUNCH: Modelling Unique 'N Controllable Heads](https://mlsanity.com/paper/2310.02753/) (2023)
- [Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generation](https://mlsanity.com/paper/2504.15259/) (2025)

## Abstract (arXiv)

We present a data-driven inference method that can synthesize a photorealistic texture map of a complete 3D face model given a partial 2D view of a person in the wild. After an initial estimation of shape and low-frequency albedo, we compute a high-frequency partial texture map, without the shading component, of the visible face area. To extract the fine appearance details from this incomplete input, we introduce a multi-scale detail analysis technique based on mid-layer feature correlations extracted from a deep convolutional neural network. We demonstrate that fitting a convex combination of feature correlations from a high-resolution face database can yield a semantically plausible facial detail description of the entire face. A complete and photorealistic texture map can then be synthesized by iteratively optimizing for the reconstructed feature correlations. Using these high-resolution textures and a commercial rendering framework, we can produce high-fidelity 3D renderings that are visually comparable to those obtained with state-of-the-art multi-view face capture systems. We demonstrate successful face reconstructions from a wide range of low resolution input images, including those of historical figures. In addition to extensive evaluations, we validate the realism of our results using a crowdsourced user study.