# Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinement

arXiv: https://arxiv.org/abs/2106.11423 · HTML: https://mlsanity.com/paper/2106.11423/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 78 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Image2StyleGAN++: How to Edit the Embedded Images?](https://mlsanity.com/paper/1911.11544/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network](https://mlsanity.com/paper/1612.04904/) (2016)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [Collaborative Learning for Faster StyleGAN Embedding](https://mlsanity.com/paper/2007.01758/) (2020)
- [Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks](https://mlsanity.com/paper/2003.05653/) (2020)

## Built on by (most cited first, 30 of 36)

- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [ClipFace: Text-guided Editing of Textured 3D Morphable Models](https://mlsanity.com/paper/2212.01406/) (2022)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [FFHQ-UV: Normalized Facial UV-Texture Dataset for 3D Face Reconstruction](https://mlsanity.com/paper/2211.13874/) (2022)
- [FitMe: Deep Photorealistic 3D Morphable Model Avatars](https://mlsanity.com/paper/2305.09641/) (2023)
- [HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field](https://mlsanity.com/paper/2309.17128/) (2023)
- [Relightify: Relightable 3D Faces from a Single Image via Diffusion Models](https://mlsanity.com/paper/2305.06077/) (2023)
- [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://mlsanity.com/paper/2203.09729/) (2022)
- [VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactment](https://mlsanity.com/paper/2312.04651/) (2023)
- [Face Generation and Editing with StyleGAN: A Survey](https://mlsanity.com/paper/2212.09102/) (2022)
- [HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details](https://mlsanity.com/paper/2303.11225/) (2023)
- [Generative AI for Film Creation: A Survey of Recent Advances](https://mlsanity.com/paper/2504.08296/) (2025)
- [Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization](https://mlsanity.com/paper/2305.03043/) (2023)
- [Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance](https://mlsanity.com/paper/2501.05379/) (2025)
- [Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar](https://mlsanity.com/paper/2307.05000/) (2023)
- [Synthetic Prior for Few-Shot Drivable Head Avatar Inversion](https://mlsanity.com/paper/2501.06903/) (2025)
- [DiffPortrait360: Consistent Portrait Diffusion for 360 View Synthesis](https://mlsanity.com/paper/2503.15667/) (2025)
- [Facial Depth and Normal Estimation using Single Dual-Pixel Camera](https://mlsanity.com/paper/2111.12928/) (2021)
- [SOAP: Style-Omniscient Animatable Portraits](https://mlsanity.com/paper/2505.05022/) (2025)
- [BareSkinNet: De-makeup and De-lighting via 3D Face Reconstruction](https://mlsanity.com/paper/2209.09029/) (2022)
- [AgileAvatar: Stylized 3D Avatar Creation via Cascaded Domain Bridging](https://mlsanity.com/paper/2211.07818/) (2022)
- [Makeup Prior Models for 3D Facial Makeup Estimation and Applications](https://mlsanity.com/paper/2403.17761/) (2024)
- [FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy](https://mlsanity.com/paper/2503.17197/) (2025)
- [FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models](https://mlsanity.com/paper/2312.04465/) (2023)
- [Monocular Identity-Conditioned Facial Reflectance Reconstruction](https://mlsanity.com/paper/2404.00301/) (2024)
- [ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Sampling](https://mlsanity.com/paper/2405.16570/) (2024)
- [Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelines](https://mlsanity.com/paper/2204.10746/) (2022)
- [TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modeling](https://mlsanity.com/paper/2505.05672/) (2025)
- [A comprehensive survey on semantic facial attribute editing using generative adversarial networks](https://mlsanity.com/paper/2205.10587/) (2022)
- [Physically-Based Face Rendering for NIR-VIS Face Recognition](https://mlsanity.com/paper/2211.06408/) (2022)

## Abstract (arXiv)

We introduce a highly robust GAN-based framework for digitizing a normalized 3D avatar of a person from a single unconstrained photo. While the input image can be of a smiling person or taken in extreme lighting conditions, our method can reliably produce a high-quality textured model of a person's face in neutral expression and skin textures under diffuse lighting condition. Cutting-edge 3D face reconstruction methods use non-linear morphable face models combined with GAN-based decoders to capture the likeness and details of a person but fail to produce neutral head models with unshaded albedo textures which is critical for creating relightable and animation-friendly avatars for integration in virtual environments. The key challenges for existing methods to work is the lack of training and ground truth data containing normalized 3D faces. We propose a two-stage approach to address this problem. First, we adopt a highly robust normalized 3D face generator by embedding a non-linear morphable face model into a StyleGAN2 network. This allows us to generate detailed but normalized facial assets. This inference is then followed by a perceptual refinement step that uses the generated assets as regularization to cope with the limited available training samples of normalized faces. We further introduce a Normalized Face Dataset, which consists of a combination photogrammetry scans, carefully selected photographs, and generated fake people with neutral expressions in diffuse lighting conditions. While our prepared dataset contains two orders of magnitude less subjects than cutting edge GAN-based 3D facial reconstruction methods, we show that it is possible to produce high-quality normalized face models for very challenging unconstrained input images, and demonstrate superior performance to the current state-of-the-art.