# Controllable Light Diffusion for Portraits

arXiv: https://arxiv.org/abs/2305.04745 · HTML: https://mlsanity.com/paper/2305.04745/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning](https://mlsanity.com/paper/2004.11660/) (2020)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis](https://mlsanity.com/paper/2110.15678/) (2021)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality](https://mlsanity.com/paper/1904.01175/) (2019)
- [Neural Video Portrait Relighting in Real-time via Consistency Modeling](https://mlsanity.com/paper/2104.00484/) (2021)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)
- [VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting](https://mlsanity.com/paper/2201.04873/) (2022)
- [Geometry-aware Single-image Full-body Human Relighting](https://mlsanity.com/paper/2207.04750/) (2022)
- [LIME: Live Intrinsic Material Estimation](https://mlsanity.com/paper/1801.01075/) (2018)
- [Learning from Synthetic Shadows for Shadow Detection and Removal](https://mlsanity.com/paper/2101.01713/) (2021)
- [RoI Tanh-polar Transformer Network for Face Parsing in the Wild](https://mlsanity.com/paper/2102.02717/) (2021)
- [Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body Human Images](https://mlsanity.com/paper/1908.02714/) (2019)
- [Learning Illumination from Diverse Portraits](https://mlsanity.com/paper/2008.02396/) (2020)
- [BareSkinNet: De-makeup and De-lighting via 3D Face Reconstruction](https://mlsanity.com/paper/2209.09029/) (2022)
- [Controllable Data Augmentation Through Deep Relighting](https://mlsanity.com/paper/2110.13996/) (2021)
- [Deep Portrait Delighting](https://mlsanity.com/paper/2203.12088/) (2022)

## Built on by (most cited first, 11 of 11)

- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis](https://mlsanity.com/paper/2409.08947/) (2024)
- [Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset](https://mlsanity.com/paper/2503.14485/) (2025)
- [Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization](https://mlsanity.com/paper/2504.03011/) (2025)
- [When Two LLMs Debate, Both Think They'll Win](https://mlsanity.com/paper/2505.19184/) (2025)
- [Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory](https://mlsanity.com/paper/2407.20785/) (2024)
- [COMPOSE: Comprehensive Portrait Shadow Editing](https://mlsanity.com/paper/2408.13922/) (2024)
- [PI-Light: Physics-Inspired Diffusion for Full-Image Relighting](https://mlsanity.com/paper/2601.22135/) (2026)
- [Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal](https://mlsanity.com/paper/2507.04692/) (2025)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models](https://mlsanity.com/paper/2605.28067/) (2026)

## Abstract (arXiv)

We introduce light diffusion, a novel method to improve lighting in portraits, softening harsh shadows and specular highlights while preserving overall scene illumination. Inspired by professional photographers' diffusers and scrims, our method softens lighting given only a single portrait photo. Previous portrait relighting approaches focus on changing the entire lighting environment, removing shadows (ignoring strong specular highlights), or removing shading entirely. In contrast, we propose a learning based method that allows us to control the amount of light diffusion and apply it on in-the-wild portraits. Additionally, we design a method to synthetically generate plausible external shadows with sub-surface scattering effects while conforming to the shape of the subject's face. Finally, we show how our approach can increase the robustness of higher level vision applications, such as albedo estimation, geometry estimation and semantic segmentation.