# SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces

arXiv: https://arxiv.org/abs/2501.09756 · HTML: https://mlsanity.com/paper/2501.09756/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 47 of its 69 references that are in the map.
- First posted on arXiv 16 Jan 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 47)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model](https://mlsanity.com/paper/2212.05034/) (2022)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [Fake It Till You Make It: Face analysis in the wild using synthetic data alone](https://mlsanity.com/paper/2109.15102/) (2021)

## Built on by (most cited first, 24 of 24)

- [UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback](https://mlsanity.com/paper/2511.01678/) (2025)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [LiveLight: Real-time Streaming Video Relighting with Interactive Control](https://mlsanity.com/paper/2608.01771/) (2026)
- [UniLight: A Unified Representation for Lighting](https://mlsanity.com/paper/2512.04267/) (2025)
- [VideoMaMa: Mask-Guided Video Matting via Generative Prior](https://mlsanity.com/paper/2601.14255/) (2026)
- [GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers](https://mlsanity.com/paper/2604.20715/) (2026)
- [TransLight: Image-Guided Customized Lighting Control with Generative Decoupling](https://mlsanity.com/paper/2508.14814/) (2025)
- [Light4D: Training-Free Extreme Viewpoint 4D Video Relighting](https://mlsanity.com/paper/2602.11769/) (2026)
- [Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning](https://mlsanity.com/paper/2605.13852/) (2026)
- [GTAvatar: Bridging Gaussian Splatting and Texture Mapping for Relightable and Editable Gaussian Avatars](https://mlsanity.com/paper/2512.09162/) (2025)
- [WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering](https://mlsanity.com/paper/2512.11237/) (2025)
- [POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling](https://mlsanity.com/paper/2512.13192/) (2025)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [FashionPose: Unified Text-Driven Fashion Synthesis with Joint Geometric and Photometric Control](https://mlsanity.com/paper/2507.13311/) (2025)
- [Using Gaussian Splats to Create High-Fidelity Facial Geometry and Texture](https://mlsanity.com/paper/2512.16397/) (2025)
- [HeadLighter: Disentangling Illumination in Generative 3D Gaussian Heads via Lightstage Captures](https://mlsanity.com/paper/2601.02103/) (2026)
- [CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control](https://mlsanity.com/paper/2603.14241/) (2026)
- [FusionRelight: Relighting Portraits in Real Time via Hybrid Domain Knowledge Fusion](https://mlsanity.com/paper/2604.23094/) (2026)
- [Lighting-aware Unified Model for Instance Segmentation](https://mlsanity.com/paper/2605.20436/) (2026)
- [BodyReLux: Temporally Consistent Full-Body Video Relighting](https://mlsanity.com/paper/2605.21766/) (2026)
- [HarmoVid: Relightful Video Portrait Harmonization](https://mlsanity.com/paper/2605.28811/) (2026)
- [Generative Relightable Avatars](https://mlsanity.com/paper/2606.22718/) (2026)
- [Consistent Feature Transport for Image Relighting](https://mlsanity.com/paper/2607.17833/) (2026)

## Abstract (arXiv)

We introduce SynthLight, a diffusion model for portrait relighting. Our approach frames image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting conditions. Using a physically-based rendering engine, we synthesize a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting. We propose two training and inference strategies to bridge the gap between the synthetic and real image domains: (1) multi-task training that takes advantage of real human portraits without lighting labels; (2) an inference time diffusion sampling procedure based on classifier-free guidance that leverages the input portrait to better preserve details. Our method generalizes to diverse real photographs and produces realistic illumination effects, including specular highlights and cast shadows, while preserving the subject's identity. Our quantitative experiments on Light Stage data demonstrate results comparable to state-of-the-art relighting methods. Our qualitative results on in-the-wild images showcase rich and unprecedented illumination effects. Project Page: urlhttps://vrroom.github.io/synthlight/