# Learning a Delighting Prior for Facial Appearance Capture in the Wild

arXiv: https://arxiv.org/abs/2605.05636 · HTML: https://mlsanity.com/paper/2605.05636/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 71 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 30 of 35)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads](https://mlsanity.com/paper/2305.03027/) (2023)
- [Learning Formation of Physically-Based Face Attributes](https://mlsanity.com/paper/2004.03458/) (2020)
- [AvatarMe: Realistically Renderable 3D Facial Reconstruction "in-the-wild"](https://mlsanity.com/paper/2003.13845/) (2020)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [A Morphable Face Albedo Model](https://mlsanity.com/paper/2004.02711/) (2020)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [Portrait Shadow Manipulation](https://mlsanity.com/paper/2005.08925/) (2020)
- [FLARE: Fast Learning of Animatable and Relightable Mesh Avatars](https://mlsanity.com/paper/2310.17519/) (2023)
- [Practical Face Reconstruction via Differentiable Ray Tracing](https://mlsanity.com/paper/2101.05356/) (2021)
- [Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures](https://mlsanity.com/paper/2410.00630/) (2024)
- [Lite2Relight: 3D-aware Single Image Portrait Relighting](https://mlsanity.com/paper/2407.10487/) (2024)
- [High-Res Facial Appearance Capture from Polarized Smartphone Images](https://mlsanity.com/paper/2212.01160/) (2022)
- [Light Stage Super-Resolution: Continuous High-Frequency Relighting](https://mlsanity.com/paper/2010.08888/) (2020)
- [MoSAR: Monocular Semi-Supervised Model for Avatar Reconstruction using Differentiable Shading](https://mlsanity.com/paper/2312.13091/) (2023)
- [Video-driven Neural Physically-based Facial Asset for Production](https://mlsanity.com/paper/2202.05592/) (2022)
- [DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows](https://mlsanity.com/paper/2304.09479/) (2023)
- [An Implicit Parametric Morphable Dental Model](https://mlsanity.com/paper/2211.11402/) (2022)
- [Generative Portrait Shadow Removal](https://mlsanity.com/paper/2410.05525/) (2024)
- [FlexAvatar: Learning Complete 3D Head Avatars with Partial Supervision](https://mlsanity.com/paper/2512.15599/) (2025)
- [High-Quality Facial Geometry and Appearance Capture at Home](https://mlsanity.com/paper/2312.03442/) (2023)
- [DreamLight: Towards Harmonious and Consistent Image Relighting](https://mlsanity.com/paper/2506.14549/) (2025)
- [3DPR: Single Image 3D Portrait Relight using Generative Priors](https://mlsanity.com/paper/2510.15846/) (2025)

## Abstract (arXiv)

High-quality facial appearance capture has traditionally required costly studio recording. Recent works consider an in-the-wild smartphone-based setup; however, their model-based inverse rendering paradigm struggles with the complex disentanglement of reflectance from unknown illumination. To bridge this gap, we propose to shift the paradigm into training a powerful delighting network as a prior to constrain the optimization. We leverage the OLAT dataset and the rendered Light Stage scans for training, and propose Dataset Latent Modulation (DLM) to seamlessly integrate these heterogeneous data sources. Specifically, by conditioning the core network on learnable source-aware tokens, we decouple dataset-specific styles from physical delighting principles, enabling the emergence of a delighting prior that outperforms existing proprietary models. This powerful delighting prior enables a simple and automatic appearance capture pipeline that achieves high-quality reflectance estimation from casual video inputs, outperforming prior arts by a large margin. Furthermore, we leverage our appearance capture method to transform the multi-view NeRSemble dataset into NeRSemble-Scan, a large-scale collection of 4K-resolution relightable scans. By open-sourcing our model and the NeRSemble-Scan dataset, we democratize high-end facial capture and provide a new foundation for the research community to build photorealistic digital humans.