mlsanityData to 14 Oct 2026

AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs

arXiv 2112.05957 ↗First posted 11 Dec 2021cs.CV cs.GRcited by 49

Not on the map. Why?

Abstract (arXiv)

Over the last years, many face analysis tasks have accomplished astounding performance, with applications including face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this work, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. We capture a large dataset of facial shape and reflectance, which we have made public. We define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image, that can be rendered in various applications, and bridge the uncanny valley.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 43 of its 91 references that are in the map. First posted on arXiv 11 Dec 2021.

Builds on

43 of 91 references.

References · 43

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  4. 2018Image Super-Resolution Using Very Deep Residual Channel Attention Networkscited by 1,264
  5. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  6. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  7. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  8. 2017Neural 3D Mesh Renderercited by 437
  9. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  10. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  11. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  12. 2020FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Predictioncited by 196
31 more
  1. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  2. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  3. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  4. 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
  5. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  6. 2018Unsupervised Training for 3D Morphable Model Regressioncited by 100
  7. 2018InverseRenderNet: Learning single image inverse renderingcited by 100
  8. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  9. 2020Learning Formation of Physically-Based Face Attributescited by 94
  10. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  11. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  12. 2018FML: Face Model Learning from Videoscited by 78
  13. 2017Neural Face Editing with Intrinsic Image Disentanglingcited by 70
  14. 2018Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Imagecited by 66
  15. 20163D Face Reconstruction by Learning from Synthetic Datacited by 64
  16. 2019Towards a complete 3D morphable model of the human headcited by 63
  17. 2017Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translationcited by 56
  18. 2018On Learning 3D Face Morphable Model from In-the-wild Imagescited by 54
  19. 2020A Morphable Face Albedo Modelcited by 54
  20. 2018Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networkscited by 51
  21. 2019Combining 3D Morphable Models: A Large scale Face-and-Head Modelcited by 49
  22. 2019Photo-Realistic Facial Details Synthesis from Single Imagecited by 40
  23. 2020OSTeC: One-Shot Texture Completioncited by 37
  24. 2021Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinementcited by 36
  25. 2021Practical Face Reconstruction via Differentiable Ray Tracingcited by 35
  26. 2019Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networkscited by 34
  27. 2017Synthesizing Normalized Faces from Facial Identity Featurescited by 33
  28. 2016Photorealistic Facial Texture Inference Using Deep Neural Networkscited by 32
  29. 2019Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoderscited by 22
  30. 2020Monocular Reconstruction of Neural Face Reflectance Fieldscited by 20
  31. 2020Deep SVBRDF Estimation on Real Materialscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 13; 2024: 9; 2025: 12; 2026: 6.

49 papers.

Sort:

By area:

  1. 2022Towards Metrical Reconstruction of Human Facescited by 166
  2. 2023Relightable Gaussian Codec Avatarscited by 127
  3. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 60
  4. 2024URAvatar: Universal Relightable Gaussian Codec Avatarscited by 43
  5. 2023FitMe: Deep Photorealistic 3D Morphable Model Avatarscited by 38
  6. 2023HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Fieldcited by 37
  7. 2023Relightify: Relightable 3D Faces from a Single Image via Diffusion Modelscited by 34
  8. 2023VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactmentcited by 33
  9. 2022Facial Geometric Detail Recovery via Implicit Representationcited by 27
  10. 2024VRMM: A Volumetric Relightable Morphable Head Modelcited by 19
  11. 2023MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shadingcited by 18
  12. 2022Video-driven Neural Physically-based Facial Asset for Productioncited by 17
37 more
  1. 2025Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidancecited by 16
  2. 2023Implicit Neural Head Synthesis via Controllable Local Deformation Fieldscited by 14
  3. 2025Synthetic Prior for Few-Shot Drivable Head Avatar Inversioncited by 14
  4. 2023Towards High-Fidelity Text-Guided 3D Face Generation and Manipulation Using only Imagescited by 12
  5. 20223DMM-RF: Convolutional Radiance Fields for 3D Face Modelingcited by 10
  6. 2022BareSkinNet: De-makeup and De-lighting via 3D Face Reconstructioncited by 10
  7. 2022Towards 3D Face Reconstruction in Perspective Projection: Estimating 6DoF Face Pose from Monocular Imagecited by 9
  8. 2023HQ3DAvatar: High Quality Controllable 3D Head Avatarcited by 9
  9. 2023My3DGen: A Scalable Personalized 3D Generative Modelcited by 8
  10. 20253DPR: Single Image 3D Portrait Relight using Generative Priorscited by 8
  11. 2024Makeup Prior Models for 3D Facial Makeup Estimation and Applicationscited by 7
  12. 2025FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategycited by 7
  13. 2025PercHead: Perceptual Head Model for Single-Image 3D Head Reconstruction & Editingcited by 6
  14. 2023FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Modelscited by 5
  15. 2024Monocular Identity-Conditioned Facial Reflectance Reconstructioncited by 5
  16. 2024ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Samplingcited by 5
  17. 2025Facial Appearance Capture at Home with Patch-Level Reflectance Priorcited by 5
  18. 2023Differentiable Shadow Mapping for Efficient Inverse Graphicscited by 4
  19. 2022Physically-Based Face Rendering for NIR-VIS Face Recognitioncited by 3
  20. 2024High-Fidelity Facial Albedo Estimation via Texture Quantizationcited by 2
  21. 2024SemUV: Deep Learning based semantic manipulation over UV texture map of virtual human headscited by 1
  22. 2024Learning to Decouple the Lights for 3D Face Texture Modelingcited by 1
  23. 2025HumanMaterial: Human Material Estimation from a Single Image via Progressive Trainingcited by 1
  24. 2025AvatarTex: High-Fidelity Facial Texture Reconstruction from Single-Image Stylized Avatarscited by 1
  25. 2025WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Renderingcited by 1
  26. 2026Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shadingcited by 1
  27. 2022Efficient Hair Style Transfer with Generative Adversarial Networkscited by 0
  28. 2023MUNCH: Modelling Unique 'N Controllable Headscited by 0
  29. 2024Text-Driven Diverse Facial Texture Generation via Progressive Latent-Space Refinementcited by 0
  30. 2025Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generationcited by 0
  31. 2025Geometry-Aware Texture Generation for 3D Head Modeling with Artist-driven Controlcited by 0
  32. 2025Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texturecited by 0
  33. 2026RelightAnyone: A Generalized Relightable 3D Gaussian Head Modelcited by 0
  34. 2026Dual Diffusion Models for Multi-modal Guided 3D Avatar Generationcited by 0
  35. 2026Learning a Delighting Prior for Facial Appearance Capture in the Wildcited by 0
  36. 2026TOPOS: High-Fidelity and Efficient Industry-Grade 3D Head Generationcited by 0
  37. 2026HAFMat: Hybrid Priors Guided Adaptive Fusion for Single-Image Human Material Estimationcited by 0