# Face Relighting with Geometrically Consistent Shadows

arXiv: https://arxiv.org/abs/2203.16681 · HTML: https://mlsanity.com/paper/2203.16681/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)
- [Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks](https://mlsanity.com/paper/2003.05653/) (2020)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)
- [Towards High Fidelity Face Relighting with Realistic Shadows](https://mlsanity.com/paper/2104.00825/) (2021)
- [Monocular Reconstruction of Neural Face Reflectance Fields](https://mlsanity.com/paper/2008.10247/) (2020)
- [Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction](https://mlsanity.com/paper/2104.00858/) (2021)
- [Efficient and Differentiable Shadow Computation for Inverse Problems](https://mlsanity.com/paper/2104.00359/) (2021)
- [Color-wise Attention Network for Low-light Image Enhancement](https://mlsanity.com/paper/1911.08681/) (2019)

## Built on by (most cited first, 30 of 43)

- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage](https://mlsanity.com/paper/2204.03648/) (2022)
- [Latent Feature-Guided Diffusion Models for Shadow Removal](https://mlsanity.com/paper/2312.02156/) (2023)
- [Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset](https://mlsanity.com/paper/2503.14485/) (2025)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [Controllable and Guided Face Synthesis for Unconstrained Face Recognition](https://mlsanity.com/paper/2207.10180/) (2022)
- [RelightableHands: Efficient Neural Relighting of Articulated Hand Models](https://mlsanity.com/paper/2302.04866/) (2023)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [Multi-View Azimuth Stereo via Tangent Space Consistency](https://mlsanity.com/paper/2303.16447/) (2023)
- [DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows](https://mlsanity.com/paper/2304.09479/) (2023)
- [ReliTalk: Relightable Talking Portrait Generation from a Single Video](https://mlsanity.com/paper/2309.02434/) (2023)
- [Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization](https://mlsanity.com/paper/2504.03011/) (2025)
- [Generative Portrait Shadow Removal](https://mlsanity.com/paper/2410.05525/) (2024)
- [FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models](https://mlsanity.com/paper/2306.00783/) (2023)
- [Personalized Video Relighting With an At-Home Light Stage](https://mlsanity.com/paper/2311.08843/) (2023)
- [Text2Relight: Creative Portrait Relighting with Text Guidance](https://mlsanity.com/paper/2412.13734/) (2024)
- [COMPOSE: Comprehensive Portrait Shadow Editing](https://mlsanity.com/paper/2408.13922/) (2024)
- [Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture](https://mlsanity.com/paper/2409.18770/) (2024)
- [EdgeRelight360: Text-Conditioned 360-Degree HDR Image Generation for Real-Time On-Device Video Portrait Relighting](https://mlsanity.com/paper/2404.09918/) (2024)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [Light Sampling Field and BRDF Representation for Physically-based Neural Rendering](https://mlsanity.com/paper/2304.05472/) (2023)
- [INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields](https://mlsanity.com/paper/2312.16197/) (2023)
- [LumiSculpt: Enabling Consistent Portrait Lighting in Video Generation](https://mlsanity.com/paper/2410.22979/) (2024)
- [All-frequency Full-body Human Image Relighting](https://mlsanity.com/paper/2411.00356/) (2024)
- [Towards Consistent and Controllable Image Synthesis for Face Editing](https://mlsanity.com/paper/2502.02465/) (2025)
- [EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization](https://mlsanity.com/paper/2603.21332/) (2026)
- [Total-Editing: Head Avatar with Editable Appearance, Motion, and Lighting](https://mlsanity.com/paper/2505.20582/) (2025)
- [DiffFAE: Advancing High-fidelity One-shot Facial Appearance Editing with Space-sensitive Customization and Semantic Preservation](https://mlsanity.com/paper/2403.17664/) (2024)

## Abstract (arXiv)

Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.