# Towards High Fidelity Face Relighting with Realistic Shadows

arXiv: https://arxiv.org/abs/2104.00825 · HTML: https://mlsanity.com/paper/2104.00825/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction](https://mlsanity.com/paper/1703.10580/) (2017)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Face Alignment in Full Pose Range: A 3D Total Solution](https://mlsanity.com/paper/1804.01005/) (2018)
- [A Closed-form Solution to Photorealistic Image Stylization](https://mlsanity.com/paper/1802.06474/) (2018)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Unsupervised Training for 3D Morphable Model Regression](https://mlsanity.com/paper/1806.06098/) (2018)
- [Towards High-fidelity Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1904.04933/) (2019)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [On Learning 3D Face Morphable Model from In-the-wild Images](https://mlsanity.com/paper/1808.09560/) (2018)
- [Towards High-Fidelity 3D Face Reconstruction from In-the-Wild Images Using Graph Convolutional Networks](https://mlsanity.com/paper/2003.05653/) (2020)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)

## Built on by (most cited first, 30 of 42)

- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [Geometry-aware Single-image Full-body Human Relighting](https://mlsanity.com/paper/2207.04750/) (2022)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [De-rendering 3D Objects in the Wild](https://mlsanity.com/paper/2201.02279/) (2022)
- [SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage](https://mlsanity.com/paper/2204.03648/) (2022)
- [Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset](https://mlsanity.com/paper/2503.14485/) (2025)
- [MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation](https://mlsanity.com/paper/2111.01048/) (2021)
- [Towards Generalizable Multi-Object Tracking](https://mlsanity.com/paper/2406.00429/) (2024)
- [RelightableHands: Efficient Neural Relighting of Articulated Hand Models](https://mlsanity.com/paper/2302.04866/) (2023)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [Adversarial Relighting Against Face Recognition](https://mlsanity.com/paper/2108.07920/) (2021)
- [DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows](https://mlsanity.com/paper/2304.09479/) (2023)
- [ReliTalk: Relightable Talking Portrait Generation from a Single Video](https://mlsanity.com/paper/2309.02434/) (2023)
- [Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization](https://mlsanity.com/paper/2504.03011/) (2025)
- [BareSkinNet: De-makeup and De-lighting via 3D Face Reconstruction](https://mlsanity.com/paper/2209.09029/) (2022)
- [Generative Portrait Shadow Removal](https://mlsanity.com/paper/2410.05525/) (2024)
- [High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model](https://mlsanity.com/paper/2502.19894/) (2025)
- [Personalized Video Relighting With an At-Home Light Stage](https://mlsanity.com/paper/2311.08843/) (2023)
- [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024)
- [Text2Relight: Creative Portrait Relighting with Text Guidance](https://mlsanity.com/paper/2412.13734/) (2024)
- [COMPOSE: Comprehensive Portrait Shadow Editing](https://mlsanity.com/paper/2408.13922/) (2024)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [Controllable Data Augmentation Through Deep Relighting](https://mlsanity.com/paper/2110.13996/) (2021)
- [Deep Portrait Delighting](https://mlsanity.com/paper/2203.12088/) (2022)
- [JoIN: Joint GANs Inversion for Intrinsic Image Decomposition](https://mlsanity.com/paper/2305.11321/) (2023)
- [LumiSculpt: Enabling Consistent Portrait Lighting in Video Generation](https://mlsanity.com/paper/2410.22979/) (2024)
- [All-frequency Full-body Human Image Relighting](https://mlsanity.com/paper/2411.00356/) (2024)

## Abstract (arXiv)

Existing face relighting methods often struggle with two problems: maintaining the local facial details of the subject and accurately removing and synthesizing shadows in the relit image, especially hard shadows. We propose a novel deep face relighting method that addresses both problems. Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting, allowing us to relight the image while maintaining the local facial details. During training, our model also learns to accurately modify shadows by using estimated shadow masks to emphasize on the high-contrast shadow borders. Furthermore, we introduce a method to use the shadow mask to estimate the ambient light intensity in an image, and are thus able to leverage multiple datasets during training with different global lighting intensities. With quantitative and qualitative evaluations on the Multi-PIE and FFHQ datasets, we demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows while achieving state-of-the-art face relighting performance.