# Faces as Lighting Probes via Unsupervised Deep Highlight Extraction

arXiv: https://arxiv.org/abs/1803.06340 · HTML: https://mlsanity.com/paper/1803.06340/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2018.

## Built on by (most cited first, 11 of 11)

- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [Learning Fine-Grained Segmentation of 3D Shapes without Part Labels](https://mlsanity.com/paper/2103.13030/) (2021)
- [Weakly-supervised Single-view Image Relighting](https://mlsanity.com/paper/2303.13852/) (2023)
- [Accidental Light Probes](https://mlsanity.com/paper/2301.05211/) (2023)
- [Cross-domain Compositing with Pretrained Diffusion Models](https://mlsanity.com/paper/2302.10167/) (2023)
- [DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpainting](https://mlsanity.com/paper/2507.01305/) (2025)
- [GCC: Generative Color Constancy via Diffusing a Color Checker](https://mlsanity.com/paper/2502.17435/) (2025)
- [Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors](https://mlsanity.com/paper/2508.08384/) (2025)
- [SAC-GAN: Structure-Aware Image Composition](https://mlsanity.com/paper/2112.06596/) (2021)
- [Relighting Scenes with Object Insertions in Neural Radiance Fields](https://mlsanity.com/paper/2406.14806/) (2024)
- [Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction](https://mlsanity.com/paper/2303.10344/) (2023)

## Abstract (arXiv)

We present a method for estimating detailed scene illumination using human faces in a single image. In contrast to previous works that estimate lighting in terms of low-order basis functions or distant point lights, our technique estimates illumination at a higher precision in the form of a non-parametric environment map. Based on the observation that faces can exhibit strong highlight reflections from a broad range of lighting directions, we propose a deep neural network for extracting highlights from faces, and then trace these reflections back to the scene to acquire the environment map. Since real training data for highlight extraction is very limited, we introduce an unsupervised scheme for finetuning the network on real images, based on the consistent diffuse chromaticity of a given face seen in multiple real images. In tracing the estimated highlights to the environment, we reduce the blurring effect of skin reflectance on reflected light through a deconvolution determined by prior knowledge on face material properties. Comparisons to previous techniques for highlight extraction and illumination estimation show the state-of-the-art performance of this approach on a variety of indoor and outdoor scenes.