# Learning Illumination from Diverse Portraits

arXiv: https://arxiv.org/abs/2008.02396 · HTML: https://mlsanity.com/paper/2008.02396/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2020.

## Built on by (most cited first, 15 of 15)

- [Relighting4D: Neural Relightable Human from Videos](https://mlsanity.com/paper/2207.07104/) (2022)
- [DifFRelight: Diffusion-Based Facial Performance Relighting](https://mlsanity.com/paper/2410.08188/) (2024)
- [A Light Stage on Every Desk](https://mlsanity.com/paper/2105.08051/) (2021)
- [Eclipse: Disambiguating Illumination and Materials using Unintended Shadows](https://mlsanity.com/paper/2305.16321/) (2023)
- [Accidental Light Probes](https://mlsanity.com/paper/2301.05211/) (2023)
- [Controllable Light Diffusion for Portraits](https://mlsanity.com/paper/2305.04745/) (2023)
- [Deep Joint Demosaicing and High Dynamic Range Imaging within a Single Shot](https://mlsanity.com/paper/2111.07281/) (2021)
- [Personalized Video Relighting With an At-Home Light Stage](https://mlsanity.com/paper/2311.08843/) (2023)
- [PhotoApp: Photorealistic Appearance Editing of Head Portraits](https://mlsanity.com/paper/2103.07658/) (2021)
- [Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality](https://mlsanity.com/paper/2301.06143/) (2023)
- [Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction](https://mlsanity.com/paper/2303.10344/) (2023)
- [A New Dimension in Testimony: Relighting Video with Reflectance Field Exemplars](https://mlsanity.com/paper/2104.02773/) (2021)
- [Lighting in Motion: Spatiotemporal HDR Lighting Estimation](https://mlsanity.com/paper/2512.13597/) (2025)
- [Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators](https://mlsanity.com/paper/2411.12002/) (2024)
- [Inpainting Normal Maps for Lightstage data](https://mlsanity.com/paper/2401.08099/) (2024)

## Abstract (arXiv)

We present a learning-based technique for estimating high dynamic range (HDR), omnidirectional illumination from a single low dynamic range (LDR) portrait image captured under arbitrary indoor or outdoor lighting conditions. We train our model using portrait photos paired with their ground truth environmental illumination. We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects in various expressions. We then relight the subjects using image-based relighting with a database of one million HDR lighting environments, compositing the relit subjects onto paired high-resolution background imagery recorded during the lighting acquisition. We train the lighting estimation model using rendering-based loss functions and add a multi-scale adversarial loss to estimate plausible high frequency lighting detail. We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation, and we also show that our method reliably handles the inherent ambiguity between overall lighting strength and surface albedo, recovering a similar scale of illumination for subjects with diverse skin tones. We demonstrate that our method allows virtual objects and digital characters to be added to a portrait photograph with consistent illumination. Our lighting inference runs in real-time on a smartphone, enabling realistic rendering and compositing of virtual objects into live video for augmented reality applications.