# AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs

arXiv: https://arxiv.org/abs/2112.05957 · HTML: https://mlsanity.com/paper/2112.05957/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 91 references that are in the map.
- First posted on arXiv 11 Dec 2021.

## Builds on (most cited first, 30 of 43)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research](https://mlsanity.com/paper/1911.05063/) (2019)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz](https://mlsanity.com/paper/1712.02859/) (2017)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [FML: Face Model Learning from Videos](https://mlsanity.com/paper/1812.07603/) (2018)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [3D Face Reconstruction by Learning from Synthetic Data](https://mlsanity.com/paper/1609.04387/) (2016)
- [Towards a complete 3D morphable model of the human head](https://mlsanity.com/paper/1911.08008/) (2019)
- [Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation](https://mlsanity.com/paper/1703.10131/) (2017)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)

## Built on by (most cited first, 30 of 49)

- [Towards Metrical Reconstruction of Human Faces](https://mlsanity.com/paper/2204.06607/) (2022)
- [Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2312.03704/) (2023)
- [ClipFace: Text-guided Editing of Textured 3D Morphable Models](https://mlsanity.com/paper/2212.01406/) (2022)
- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field](https://mlsanity.com/paper/2309.17128/) (2023)
- [Relightify: Relightable 3D Faces from a Single Image via Diffusion Models](https://mlsanity.com/paper/2305.06077/) (2023)
- [VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment](https://mlsanity.com/paper/2312.04651/) (2023)
- [Facial Geometric Detail Recovery via Implicit Representation](https://mlsanity.com/paper/2203.09692/) (2022)
- [VRMM: A Volumetric Relightable Morphable Head Model](https://mlsanity.com/paper/2402.04101/) (2024)
- [MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading](https://mlsanity.com/paper/2312.13091/) (2023)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance](https://mlsanity.com/paper/2501.05379/) (2025)
- [Implicit Neural Head Synthesis via Controllable Local Deformation Fields](https://mlsanity.com/paper/2304.11113/) (2023)
- [Synthetic Prior for Few-Shot Drivable Head Avatar Inversion](https://mlsanity.com/paper/2501.06903/) (2025)
- [Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Images](https://mlsanity.com/paper/2308.16758/) (2023)
- [3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling](https://mlsanity.com/paper/2209.07366/) (2022)
- [BareSkinNet: De-makeup and De-lighting via 3D Face Reconstruction](https://mlsanity.com/paper/2209.09029/) (2022)
- [Towards 3D Face Reconstruction in Perspective Projection: Estimating 6DoF Face Pose from Monocular Image](https://mlsanity.com/paper/2205.04126/) (2022)
- [HQ3DAvatar: High Quality Controllable 3D Head Avatar](https://mlsanity.com/paper/2303.14471/) (2023)
- [My3DGen: A Scalable Personalized 3D Generative Model](https://mlsanity.com/paper/2307.05468/) (2023)
- [3DPR: Single Image 3D Portrait Relight using Generative Priors](https://mlsanity.com/paper/2510.15846/) (2025)
- [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editing](https://mlsanity.com/paper/2511.02777/) (2025)
- [FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models](https://mlsanity.com/paper/2312.04465/) (2023)
- [Monocular Identity-Conditioned Facial Reflectance Reconstruction](https://mlsanity.com/paper/2404.00301/) (2024)
- [ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling](https://mlsanity.com/paper/2405.16570/) (2024)
- [Facial Appearance Capture at Home with Patch-Level Reflectance Prior](https://mlsanity.com/paper/2506.03478/) (2025)
- [Differentiable Shadow Mapping for Efficient Inverse Graphics](https://mlsanity.com/paper/2308.10896/) (2023)

## Abstract (arXiv)

Over the last years, many face analysis tasks have accomplished astounding performance, with applications including face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this work, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. We capture a large dataset of facial shape and reflectance, which we have made public. We define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image, that can be rendered in various applications, and bridge the uncanny valley.