mlsanityData to 7 Oct 2026

FitMe: Deep Photorealistic 3D Morphable Model Avatars

arXiv 2305.09641 ↗First posted 16 May 2023cs.CV cs.GR cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce FitMe, a facial reflectance model and a differentiable rendering optimization pipeline, that can be used to acquire high-fidelity renderable human avatars from single or multiple images. The model consists of a multi-modal style-based generator, that captures facial appearance in terms of diffuse and specular reflectance, and a PCA-based shape model. We employ a fast differentiable rendering process that can be used in an optimization pipeline, while also achieving photorealistic facial shading. Our optimization process accurately captures both the facial reflectance and shape in high-detail, by exploiting the expressivity of the style-based latent representation and of our shape model. FitMe achieves state-of-the-art reflectance acquisition and identity preservation on single "in-the-wild" facial images, while it produces impressive scan-like results, when given multiple unconstrained facial images pertaining to the same identity. In contrast with recent implicit avatar reconstructions, FitMe requires only one minute and produces relightable mesh and texture-based avatars, that can be used by end-user applications.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 45 of its 78 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

45 of 78 references.

References · 45

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  8. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  9. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  10. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  11. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  12. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
33 more
  1. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  2. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  3. 2020FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Predictioncited by 196
  4. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  5. 2018Nonlinear 3D Face Morphable Modelcited by 157
  6. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  7. 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
  8. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  9. 2018Unsupervised Training for 3D Morphable Model Regressioncited by 99
  10. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  11. 2020Learning Formation of Physically-Based Face Attributescited by 94
  12. 2019Towards High-fidelity Nonlinear 3D Face Morphable Modelcited by 91
  13. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  14. 2017Neural Face Editing with Intrinsic Image Disentanglingcited by 70
  15. 2019Towards a complete 3D morphable model of the human headcited by 63
  16. 2020A Morphable Face Albedo Modelcited by 54
  17. 2019Combining 3D Morphable Models: A Large scale Face-and-Head Modelcited by 49
  18. 2021AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANscited by 49
  19. 2021Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracingcited by 48
  20. 2020High-Fidelity 3D Digital Human Head Creation from RGB-D Selfiescited by 47
  21. 20173D Face Morphable Models "In-the-Wild"cited by 44
  22. 2019Photo-Realistic Facial Details Synthesis from Single Imagecited by 40
  23. 2020OSTeC: One-Shot Texture Completioncited by 37
  24. 2021Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinementcited by 36
  25. 2021Practical Face Reconstruction via Differentiable Ray Tracingcited by 35
  26. 2019Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networkscited by 34
  27. 2021Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstructioncited by 33
  28. 2016Photorealistic Facial Texture Inference Using Deep Neural Networkscited by 32
  29. 2022Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguationcited by 32
  30. 20193DFaceGAN: Adversarial Nets for 3D Face Representation, Generation, and Translationcited by 27
  31. 20213D human tongue reconstruction from single "in-the-wild" imagescited by 5
  32. 2022Physically-Based Face Rendering for NIR-VIS Face Recognitioncited by 3
  33. 2022Estimation of Spectral Biophysical Skin Properties from Captured RGB Albedocited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 13; 2025: 11; 2026: 6.

38 papers.

Sort:

By area:

  1. 2024SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesiscited by 81
  2. 2023MonoNPHM: Dynamic Head Reconstruction from Monocular Videoscited by 35
  3. 2023VOODOO 3D: Volumetric Portrait Disentanglement for One-Shot 3D Head Reenactmentcited by 33
  4. 2023MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shadingcited by 18
  5. 2025Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidancecited by 16
  6. 2025Synthetic Prior for Few-Shot Drivable Head Avatar Inversioncited by 14
  7. 2024UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Texturescited by 13
  8. 2024Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusioncited by 12
  9. 2025LVFace: Progressive Cluster Optimization for Large Vision Models in Face Recognitioncited by 12
  10. 2023High-Quality Facial Geometry and Appearance Capture at Homecited by 9
  11. 2024Topo4D: Topology-Preserving Gaussian Splatting for High-Fidelity 4D Head Capturecited by 8
  12. 2024Make-It-Vivid: Dressing Your Animatable Biped Cartoon Characters from Textcited by 7
26 more
  1. 2024Makeup Prior Models for 3D Facial Makeup Estimation and Applicationscited by 7
  2. 2025FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategycited by 7
  3. 2025FastAvatar: Instant 3D Gaussian Splatting for Faces from Single Unconstrained Posescited by 7
  4. 2024BodyShapeGPT: SMPL Body Shape Manipulation with LLMscited by 6
  5. 2023FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Modelscited by 5
  6. 2024Locally Adaptive Neural 3D Morphable Modelscited by 5
  7. 2024Monocular Identity-Conditioned Facial Reflectance Reconstructioncited by 5
  8. 2024ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Samplingcited by 5
  9. 2024S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketchcited by 5
  10. 2023SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Imagescited by 4
  11. 2025TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modelingcited by 4
  12. 2025SpinMeRound: Consistent Multi-View Identity Generation Using Diffusion Modelscited by 3
  13. 2024High-Fidelity Facial Albedo Estimation via Texture Quantizationcited by 2
  14. 2024Learning to Decouple the Lights for 3D Face Texture Modelingcited by 1
  15. 2025AvatarTex: High-Fidelity Facial Texture Reconstruction from Single-Image Stylized Avatarscited by 1
  16. 2025WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Renderingcited by 1
  17. 2026Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shadingcited by 1
  18. 2023MUNCH: Modelling Unique 'N Controllable Headscited by 0
  19. 2023Talking Head(?) Anime from a Single Image 4: Improved Model and Its Distillationcited by 0
  20. 2025Geometry-Aware Texture Generation for 3D Head Modeling with Artist-driven Controlcited by 0
  21. 2025BabyFlow: 3D modeling of realistic and expressive infant facescited by 0
  22. 2026Dual Diffusion Models for Multi-modal Guided 3D Avatar Generationcited by 0
  23. 2026Wispy to Voluminous: Prior-free Multi-view Capture of Strand-level Facial Haircited by 0
  24. 2026SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstructioncited by 0
  25. 2026FiCA: Feed-forward instant Gaussian Codec Avatars from a Single Portrait Imagecited by 0
  26. 2026Enhancing Facial Expression Recognition in Head-Mounted Displays with Synthetic Datacited by 0