# LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting

arXiv: https://arxiv.org/abs/2412.00177 · HTML: https://mlsanity.com/paper/2412.00177/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 59 of its 81 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 59)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations](https://mlsanity.com/paper/2410.10792/) (2024)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination](https://mlsanity.com/paper/2106.01970/) (2021)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation](https://mlsanity.com/paper/2402.11929/) (2024)
- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [Neural Light Transport for Relighting and View Synthesis](https://mlsanity.com/paper/2008.03806/) (2020)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination](https://mlsanity.com/paper/2404.11593/) (2024)
- [DiffusionLight: Light Probes for Free by Painting a Chrome Ball](https://mlsanity.com/paper/2312.09168/) (2023)
- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for now](https://mlsanity.com/paper/2311.17138/) (2023)
- [Zero-Reference Low-Light Enhancement via Physical Quadruple Priors](https://mlsanity.com/paper/2403.12933/) (2024)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [FlashTex: Fast Relightable Mesh Texturing with LightControlNet](https://mlsanity.com/paper/2402.13251/) (2024)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)

## Built on by (most cited first, 30 of 38)

- [DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models](https://mlsanity.com/paper/2501.18590/) (2025)
- [UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting](https://mlsanity.com/paper/2506.15673/) (2025)
- [LightLab: Controlling Light Sources in Images with Diffusion Models](https://mlsanity.com/paper/2505.09608/) (2025)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [Physically Controllable Relighting of Photographs](https://mlsanity.com/paper/2508.05626/) (2025)
- [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://mlsanity.com/paper/2503.21770/) (2025)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)
- [PractiLight: Practical Light Control Using Foundational Diffusion Models](https://mlsanity.com/paper/2509.01837/) (2025)
- [FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement](https://mlsanity.com/paper/2511.18346/) (2025)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [GR3EN: Generative Relighting for 3D Environments](https://mlsanity.com/paper/2601.16272/) (2026)
- [UniLight: A Unified Representation for Lighting](https://mlsanity.com/paper/2512.04267/) (2025)
- [Generative Blocks World: Moving Things Around in Pictures](https://mlsanity.com/paper/2506.20703/) (2025)
- [Principia: Relational Physics Tests for Video Models](https://mlsanity.com/paper/2609.04200/) (2026)
- [POLAR: A Portrait OLAT Dataset and Generative Framework for Illumination-Aware Face Modeling](https://mlsanity.com/paper/2512.13192/) (2025)
- [MVID: Feed-Forward Multi-View Intrinsic Image Decomposition](https://mlsanity.com/paper/2512.23667/) (2025)
- [Learning Latent Proxies for Controllable Single-Image Relighting](https://mlsanity.com/paper/2603.15555/) (2026)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relighting](https://mlsanity.com/paper/2607.08879/) (2026)
- [HeadLighter: Disentangling Illumination in Generative 3D Gaussian Heads via Lightstage Captures](https://mlsanity.com/paper/2601.02103/) (2026)
- [SyncLight: Single-Edit Multi-View Relighting](https://mlsanity.com/paper/2601.16981/) (2026)
- [Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics](https://mlsanity.com/paper/2602.01391/) (2026)
- [Adaptive Illumination Control for Robot Perception](https://mlsanity.com/paper/2602.15900/) (2026)
- [CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control](https://mlsanity.com/paper/2603.14241/) (2026)
- [LightMover: Generative Light Movement with Color and Intensity Controls](https://mlsanity.com/paper/2603.27209/) (2026)
- [PIXLRelight: Controllable Relighting via Intrinsic Conditioning](https://mlsanity.com/paper/2605.18735/) (2026)
- [TRON: Tracing Rays to Orchestrate a Neural Renderer for 3D Gaussian Reconstructions](https://mlsanity.com/paper/2606.11314/) (2026)
- [From Sparse and Imperfect 2D Anchors to Consistent 3D Gaussian Street Scenes: Support-Aware Appearance](https://mlsanity.com/paper/2606.26007/) (2026)

## Abstract (arXiv)

We introduce LumiNet, a novel architecture that leverages generative models and latent intrinsic representations for effective lighting transfer. Given a source image and a target lighting image, LumiNet synthesizes a relit version of the source scene that captures the target's lighting. Our approach makes two key contributions: a data curation strategy from the StyleGAN-based relighting model for our training, and a modified diffusion-based ControlNet that processes both latent intrinsic properties from the source image and latent extrinsic properties from the target image. We further improve lighting transfer through a learned adaptor (MLP) that injects the target's latent extrinsic properties via cross-attention and fine-tuning. Unlike traditional ControlNet, which generates images with conditional maps from a single scene, LumiNet processes latent representations from two different images - preserving geometry and albedo from the source while transferring lighting characteristics from the target. Experiments demonstrate that our method successfully transfers complex lighting phenomena including specular highlights and indirect illumination across scenes with varying spatial layouts and materials, outperforming existing approaches on challenging indoor scenes using only images as input.