# Latent Intrinsics Emerge from Training to Relight

arXiv: https://arxiv.org/abs/2405.21074 · HTML: https://mlsanity.com/paper/2405.21074/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 30 of 34)

- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [The Fast Bilateral Solver](https://mlsanity.com/paper/1511.03296/) (2015)
- [CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering](https://mlsanity.com/paper/1808.08601/) (2018)
- [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019)
- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets](https://mlsanity.com/paper/2007.12868/) (2020)
- [Fast Spatially-Varying Indoor Lighting Estimation](https://mlsanity.com/paper/1906.03799/) (2019)
- [Rendering Synthetic Objects into Legacy Photographs](https://mlsanity.com/paper/1912.11565/) (2019)
- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)
- [A Dataset of Multi-Illumination Images in the Wild](https://mlsanity.com/paper/1910.08131/) (2019)
- [StyleGAN knows Normal, Depth, Albedo, and More](https://mlsanity.com/paper/2306.00987/) (2023)
- [Self-Supervised Intrinsic Image Decomposition](https://mlsanity.com/paper/1711.03678/) (2017)
- [Automatic Scene Inference for 3D Object Compositing](https://mlsanity.com/paper/1912.12297/) (2019)
- [Learning to Factorize and Relight a City](https://mlsanity.com/paper/2008.02796/) (2020)
- [A Light Stage on Every Desk](https://mlsanity.com/paper/2105.08051/) (2021)
- [Reflectance Adaptive Filtering Improves Intrinsic Image Estimation](https://mlsanity.com/paper/1612.05062/) (2016)
- [Intrinsic Image Decomposition using Paradigms](https://mlsanity.com/paper/2011.10512/) (2020)
- [Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshading](https://mlsanity.com/paper/2010.05907/) (2020)
- [Editable Indoor Lighting Estimation](https://mlsanity.com/paper/2211.03928/) (2022)

## Built on by (most cited first, 17 of 17)

- [LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting](https://mlsanity.com/paper/2412.00177/) (2024)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [IntrinsicEdit: Precise generative image manipulation in intrinsic space](https://mlsanity.com/paper/2505.08889/) (2025)
- [Physically Controllable Relighting of Photographs](https://mlsanity.com/paper/2508.05626/) (2025)
- [PractiLight: Practical Light Control Using Foundational Diffusion Models](https://mlsanity.com/paper/2509.01837/) (2025)
- [GR3EN: Generative Relighting for 3D Environments](https://mlsanity.com/paper/2601.16272/) (2026)
- [TransLight: Image-Guided Customized Lighting Control with Generative Decoupling](https://mlsanity.com/paper/2508.14814/) (2025)
- [SAIL: Self-supervised Albedo Estimation from Real Images with a Latent Diffusion Model](https://mlsanity.com/paper/2505.19751/) (2025)
- [Generalizable Sparse-View 3D Reconstruction from Unconstrained Images](https://mlsanity.com/paper/2604.28193/) (2026)
- [SyncLight: Single-Edit Multi-View Relighting](https://mlsanity.com/paper/2601.16981/) (2026)
- [Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics](https://mlsanity.com/paper/2602.01391/) (2026)
- [Adaptive Illumination Control for Robot Perception](https://mlsanity.com/paper/2602.15900/) (2026)
- [BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering](https://mlsanity.com/paper/2606.17049/) (2026)
- [Curriculum-guided Change Detection Training: Toward Accurate Serac Fall Monitoring](https://mlsanity.com/paper/2606.28012/) (2026)
- [FreeLit: Paired-Free Indoor Relighting via Physics-Guided Diffusion](https://mlsanity.com/paper/2607.13656/) (2026)
- [Consistent Feature Transport for Image Relighting](https://mlsanity.com/paper/2607.17833/) (2026)

## Abstract (arXiv)

Image relighting is the task of showing what a scene from a source image would look like if illuminated differently. Inverse graphics schemes recover an explicit representation of geometry and a set of chosen intrinsics, then relight with some form of renderer. However error control for inverse graphics is difficult, and inverse graphics methods can represent only the effects of the chosen intrinsics. This paper describes a relighting method that is entirely data-driven, where intrinsics and lighting are each represented as latent variables. Our approach produces SOTA relightings of real scenes, as measured by standard metrics. We show that albedo can be recovered from our latent intrinsics without using any example albedos, and that the albedos recovered are competitive with SOTA methods.