mlsanityData to 7 Oct 2026

Learning a 3D Morphable Face Reflectance Model from Low-cost Data

arXiv 2303.11686 ↗First posted 21 Mar 2023cs.CV cs.GRcited by 20

Not on the map. Why?

Abstract (arXiv)

Modeling non-Lambertian effects such as facial specularity leads to a more realistic 3D Morphable Face Model. Existing works build parametric models for diffuse and specular albedo using Light Stage data. However, only diffuse and specular albedo cannot determine the full BRDF. In addition, the requirement of Light Stage data is hard to fulfill for the research communities. This paper proposes the first 3D morphable face reflectance model with spatially varying BRDF using only low-cost publicly-available data. We apply linear shiness weighting into parametric modeling to represent spatially varying specular intensity and shiness. Then an inverse rendering algorithm is developed to reconstruct the reflectance parameters from non-Light Stage data, which are used to train an initial morphable reflectance model. To enhance the model's generalization capability and expressive power, we further propose an update-by-reconstruction strategy to finetune it on an in-the-wild dataset. Experimental results show that our method obtains decent rendering results with plausible facial specularities. Our code is released hrefhttps://yxuhan.github.io/ReflectanceMM/index.htmltextcolormagentahere.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 45 of its 84 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

45 of 84 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  8. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 911
  9. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  10. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  11. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  12. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
33 more
  1. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 481
  2. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  3. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  4. 2020BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentationcited by 294
  5. 20193D Morphable Face Models -- Past, Present and Futurecited by 269
  6. 2020StyleRig: Rigging StyleGAN for 3D Control over Portrait Imagescited by 267
  7. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  8. 2021I M Avatar: Implicit Morphable Head Avatars from Videoscited by 230
  9. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  10. 2020FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Predictioncited by 196
  11. 2019Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wildcited by 180
  12. 2019Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervisioncited by 176
  13. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  14. 2018Nonlinear 3D Face Morphable Modelcited by 157
  15. 2017Morphable Face Models - An Open Frameworkcited by 138
  16. 2018Unsupervised Training for 3D Morphable Model Regressioncited by 99
  17. 2020Learning Formation of Physically-Based Face Attributescited by 94
  18. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  19. 2019Towards High-fidelity Nonlinear 3D Face Morphable Modelcited by 91
  20. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  21. 2018FML: Face Model Learning from Videoscited by 78
  22. 2020A Morphable Face Albedo Modelcited by 54
  23. 2021Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracingcited by 48
  24. 2020Learning Complete 3D Morphable Face Models from Images and Videoscited by 35
  25. 2021Practical Face Reconstruction via Differentiable Ray Tracingcited by 35
  26. 2021Riggable 3D Face Reconstruction via In-Network Optimizationcited by 31
  27. 2022SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stagecited by 30
  28. 2020Personalized Face Modeling for Improved Face Reconstruction and Motion Retargetingcited by 27
  29. 2021A Light Stage on Every Deskcited by 21
  30. 2022High-Res Facial Appearance Capture from Polarized Smartphone Imagescited by 21
  31. 2020Monocular Reconstruction of Neural Face Reflectance Fieldscited by 20
  32. 2022S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Imagecited by 17
  33. 2020Relightable 3D Head Portraits from a Smartphone Videocited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 10; 2025: 2; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generationcited by 93
  2. 2024Neural Gaffer: Relighting Any Object via Diffusioncited by 91
  3. 2023High-Quality Facial Geometry and Appearance Capture at Homecited by 9
  4. 2023FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Modelscited by 5
  5. 2024Monocular Identity-Conditioned Facial Reflectance Reconstructioncited by 5
  6. 2023SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Imagescited by 4
  7. 2023Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identificationcited by 2
  8. 2024Monocular Facial Appearance Capture in the Wildcited by 2
  9. 2025WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Renderingcited by 1
  10. 2023Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adaptercited by 0
  11. 2023GMTR: Graph Matching Transformerscited by 0
  12. 2023DarkShot: Lighting Dark Images with Low-Compute and High-Qualitycited by 0
8 more
  1. 2024TRLS: A Time Series Representation Learning Framework via Spectrogram for Medical Signal Processingcited by 0
  2. 2024TEN-GUARD: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networkscited by 0
  3. 2024T-Pixel2Mesh: Combining Global and Local Transformer for 3D Mesh Generation from a Single Imagecited by 0
  4. 2024Strategic Arms with Side Communication Prevail Over Low-Regret MAB Algorithmscited by 0
  5. 2024Towards General Deepfake Detection with Dynamic Curriculumcited by 0
  6. 2024Attr-Int: A Simple and Effective Entity Alignment Framework for Heterogeneous Knowledge Graphscited by 0
  7. 2025Mask6D: Masked Pose Priors For 6D Object Pose Estimationcited by 0
  8. 2026Light Up Your Face: A Physically Consistent Dataset and Diffusion Model for Face Fill-Light Enhancementcited by 0