mlsanityData to 14 Oct 2026

Relightify: Relightable 3D Faces from a Single Image via Diffusion Models

arXiv 2305.06077 ↗First posted 10 May 2023cs.CVcited by 34

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Following the remarkable success of diffusion models on image generation, recent works have also demonstrated their impressive ability to address a number of inverse problems in an unsupervised way, by properly constraining the sampling process based on a conditioning input. Motivated by this, in this paper, we present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. We start by leveraging a high-quality UV dataset of facial reflectance (diffuse and specular albedo and normals), which we render under varying illumination settings to simulate natural RGB textures and, then, train an unconditional diffusion model on concatenated pairs of rendered textures and reflectance components. At test time, we fit a 3D morphable model to the given image and unwrap the face in a partial UV texture. By sampling from the diffusion model, while retaining the observed texture part intact, the model inpaints not only the self-occluded areas but also the unknown reflectance components, in a single sequence of denoising steps. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent reflectance estimation. Through a series of qualitative and quantitative comparisons, we demonstrate superior performance in both texture completion as well as reflectance reconstruction tasks.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 44 of its 70 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

44 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 43

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
31 more
  1. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  2. 2022Denoising Diffusion Restoration Modelscited by 963
  3. 2021Score-based Generative Modeling in Latent Spacecited by 636
  4. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  5. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 632
  6. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 508
  7. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  8. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 307
  9. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  10. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
  11. 2019GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstructioncited by 150
  12. 2016Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Networkcited by 109
  13. 2018Unsupervised Training for 3D Morphable Model Regressioncited by 100
  14. 2020Learning Formation of Physically-Based Face Attributescited by 94
  15. 2021Noise Estimation for Generative Diffusion Modelscited by 94
  16. 2019Towards High-fidelity Nonlinear 3D Face Morphable Modelcited by 91
  17. 2020AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"cited by 91
  18. 2017Generative Face Completioncited by 79
  19. 2017UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognitioncited by 62
  20. 2020A Morphable Face Albedo Modelcited by 54
  21. 2021AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANscited by 49
  22. 2021Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracingcited by 48
  23. 2020High-Fidelity 3D Digital Human Head Creation from RGB-D Selfiescited by 47
  24. 2019Photo-Realistic Facial Details Synthesis from Single Imagecited by 40
  25. 2020OSTeC: One-Shot Texture Completioncited by 37
  26. 2021Normalized Avatar Synthesis Using StyleGAN and Perceptual Refinementcited by 36
  27. 2021Practical Face Reconstruction via Differentiable Ray Tracingcited by 35
  28. 2019Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networkscited by 34
  29. 2021Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstructioncited by 33
  30. 2016Photorealistic Facial Texture Inference Using Deep Neural Networkscited by 32
  31. 2022Towards Racially Unbiased Skin Tone Estimation via Scene Disambiguationcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 13; 2025: 10; 2026: 4.

34 papers.

Sort:

By area:

  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2023Relightable Gaussian Codec Avatarscited by 127
  3. 2024Arc2Face: A Foundation Model for ID-Consistent Human Facescited by 101
  4. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  5. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  6. 2023MonoNPHM: Dynamic Head Reconstruction from Monocular Videoscited by 35
  7. 2024Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Imagecited by 32
  8. 2024Real-time 3D-aware Portrait Video Relightingcited by 21
  9. 2024VRMM: A Volumetric Relightable Morphable Head Modelcited by 19
  10. 2023MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shadingcited by 18
  11. 2025Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidancecited by 16
  12. 2025Synthetic Prior for Few-Shot Drivable Head Avatar Inversioncited by 14
22 more
  1. 2024UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Texturescited by 13
  2. 2023High-Quality Facial Geometry and Appearance Capture at Homecited by 9
  3. 2025FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategycited by 7
  4. 2023FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Modelscited by 5
  5. 2024Locally Adaptive Neural 3D Morphable Modelscited by 5
  6. 2024Monocular Identity-Conditioned Facial Reflectance Reconstructioncited by 5
  7. 2024ID-to-3D: Expressive ID-guided 3D Heads via Score Distillation Samplingcited by 5
  8. 2025Facial Appearance Capture at Home with Patch-Level Reflectance Priorcited by 5
  9. 2026PI-Light: Physics-Inspired Diffusion for Full-Image Relightingcited by 5
  10. 2023SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Imagescited by 4
  11. 2024A Comprehensive Survey on Diffusion Models and Their Applicationscited by 4
  12. 2025TeGA: Texture Space Gaussian Avatars for High-Resolution Dynamic Head Modelingcited by 4
  13. 2024MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representationcited by 3
  14. 2025SpinMeRound: Consistent Multi-View Identity Generation Using Diffusion Modelscited by 3
  15. 2024High-Fidelity Facial Albedo Estimation via Texture Quantizationcited by 2
  16. 2025High-Quality Facial Albedo Generation for 3D Face Reconstruction from a Single Image using a Coarse-to-Fine Approachcited by 2
  17. 2025Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenescited by 1
  18. 2025HumanMaterial: Human Material Estimation from a Single Image via Progressive Trainingcited by 1
  19. 2026Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shadingcited by 1
  20. 2025Geometry-Aware Texture Generation for 3D Head Modeling with Artist-driven Controlcited by 0
  21. 2026Wispy to Voluminous: Prior-free Multi-view Capture of Strand-level Facial Haircited by 0
  22. 2026HAFMat: Hybrid Priors Guided Adaptive Fusion for Single-Image Human Material Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows
  2. 2026DIAMOND-SSS: Diffusion-Augmented Multi-View Optimization for Data-efficient SubSurface Scattering
  3. 2023Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance
  4. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
  5. 2026OMGTex: One-stage Multi-style Facial Texture Reconstruction without Geometry Guidance
  6. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
  7. 2024Coherent 3D Scene Diffusion From a Single RGB Image
  8. 2025SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces