# SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage

arXiv: https://arxiv.org/abs/2204.03648 · HTML: https://mlsanity.com/paper/2204.03648/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 69 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Nonlinear 3D Face Morphable Model](https://mlsanity.com/paper/1804.03786/) (2018)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Neural Light Transport for Relighting and View Synthesis](https://mlsanity.com/paper/2008.03806/) (2020)
- [Combining 3D Morphable Models: A Large scale Face-and-Head Model](https://mlsanity.com/paper/1903.03785/) (2019)
- [Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing](https://mlsanity.com/paper/2103.15432/) (2021)
- [Neural Video Portrait Relighting in Real-time via Consistency Modeling](https://mlsanity.com/paper/2104.00484/) (2021)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)
- [Towards High Fidelity Face Relighting with Realistic Shadows](https://mlsanity.com/paper/2104.00825/) (2021)
- [Learning to Factorize and Relight a City](https://mlsanity.com/paper/2008.02796/) (2020)
- [Multilinear Wavelets: A Statistical Shape Space for Human Faces](https://mlsanity.com/paper/1401.2818/) (2014)
- [A Light Stage on Every Desk](https://mlsanity.com/paper/2105.08051/) (2021)
- [Light Stage Super-Resolution: Continuous High-Frequency Relighting](https://mlsanity.com/paper/2010.08888/) (2020)

## Built on by (most cited first, 30 of 30)

- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos](https://mlsanity.com/paper/2407.08414/) (2024)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset](https://mlsanity.com/paper/2503.14485/) (2025)
- [SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces](https://mlsanity.com/paper/2501.09756/) (2025)
- [High-Res Facial Appearance Capture from Polarized Smartphone Images](https://mlsanity.com/paper/2212.01160/) (2022)
- [Learning a 3D Morphable Face Reflectance Model from Low-cost Data](https://mlsanity.com/paper/2303.11686/) (2023)
- [URHand: Universal Relightable Hands](https://mlsanity.com/paper/2401.05334/) (2024)
- [LumiGAN: Unconditional Generation of Relightable 3D Human Faces](https://mlsanity.com/paper/2304.13153/) (2023)
- [ReliTalk: Relightable Talking Portrait Generation from a Single Video](https://mlsanity.com/paper/2309.02434/) (2023)
- [Relightable Neural Actor with Intrinsic Decomposition and Pose Control](https://mlsanity.com/paper/2312.11587/) (2023)
- [TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer](https://mlsanity.com/paper/2506.18904/) (2025)
- [High-Quality Facial Geometry and Appearance Capture at Home](https://mlsanity.com/paper/2312.03442/) (2023)
- [Personalized Video Relighting With an At-Home Light Stage](https://mlsanity.com/paper/2311.08843/) (2023)
- [Text2Relight: Creative Portrait Relighting with Text Guidance](https://mlsanity.com/paper/2412.13734/) (2024)
- [IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation](https://mlsanity.com/paper/2506.03150/) (2025)
- [COMPOSE: Comprehensive Portrait Shadow Editing](https://mlsanity.com/paper/2408.13922/) (2024)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images](https://mlsanity.com/paper/2312.06085/) (2023)
- [LumiSculpt: Enabling Consistent Portrait Lighting in Video Generation](https://mlsanity.com/paper/2410.22979/) (2024)
- [Monocular Facial Appearance Capture in the Wild](https://mlsanity.com/paper/2412.12765/) (2024)
- [Refracting Reality: Generating Images with Realistic Transparent Objects](https://mlsanity.com/paper/2511.17340/) (2025)
- [WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering](https://mlsanity.com/paper/2512.11237/) (2025)
- [KaoLRM: Repurposing Pre-trained Large Reconstruction Models for Parametric 3D Face Reconstruction](https://mlsanity.com/paper/2601.12736/) (2026)
- [Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture](https://mlsanity.com/paper/2512.16397/) (2025)
- [RelightAnyone: A Generalized Relightable 3D Gaussian Head Model](https://mlsanity.com/paper/2601.03357/) (2026)
- [Hi-Light: A Path to high-fidelity, high-resolution video relighting with a Novel Evaluation Paradigm](https://mlsanity.com/paper/2601.23167/) (2026)

## Abstract (arXiv)

A light stage uses a series of calibrated cameras and lights to capture a subject's facial appearance under varying illumination and viewpoint. This captured information is crucial for facial reconstruction and relighting. Unfortunately, light stages are often inaccessible: they are expensive and require significant technical expertise for construction and operation. In this paper, we present SunStage: a lightweight alternative to a light stage that captures comparable data using only a smartphone camera and the sun. Our method only requires the user to capture a selfie video outdoors, rotating in place, and uses the varying angles between the sun and the face as guidance in joint reconstruction of facial geometry, reflectance, camera pose, and lighting parameters. Despite the in-the-wild un-calibrated setting, our approach is able to reconstruct detailed facial appearance and geometry, enabling compelling effects such as relighting, novel view synthesis, and reflectance editing. Results and interactive demos are available at https://sunstage.cs.washington.edu/.