mlsanityData to 14 Oct 2026

LightIt: Illumination Modeling and Control for Diffusion Models

arXiv 2403.10615 ↗First posted 15 Mar 2024cs.CV cs.GR cs.LGcited by 57

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 37 of its 52 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

37 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal9 Dec

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  12. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
24 more
  1. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  2. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  3. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  4. 2022On the detection of synthetic images generated by diffusion modelscited by 283
  5. 2020State of the Art on Neural Renderingcited by 278
  6. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  7. 2019DIODE: A Dense Indoor and Outdoor DEpth Datasetcited by 240
  8. 2016DeMoN: Depth and Motion Network for Learning Monocular Stereocited by 201
  9. 2017Neural Style Transfer: A Reviewcited by 199
  10. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  11. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 174
  12. 2017Deep Photo Style Transfercited by 168
  13. 2019Single Image Portrait Relightingcited by 152
  14. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  15. 2021NeRF for Outdoor Scene Relightingcited by 114
  16. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 90
  17. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 77
  18. 2019Deep Sky Modeling for Single Image Outdoor Lighting Estimationcited by 66
  19. 2021Free-viewpoint Indoor Neural Relighting from Multi-view Stereocited by 50
  20. 2019Learning Single Camera Depth Estimation using Dual-Pixelscited by 49
  21. 2021Self-supervised Outdoor Scene Relightingcited by 48
  22. 2022OutCast: Outdoor Single-image Relighting with Cast Shadowscited by 30
  23. 2021A Light Stage on Every Deskcited by 21
  24. 2022SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesiscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 22; 2026: 22.

57 papers.

Sort:

By area:

  1. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 57
  2. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 40
  3. 2024LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relightingcited by 38
  4. 2025LightLab: Controlling Light Sources in Images with Diffusion Modelscited by 38
  5. 2025LBM: Latent Bridge Matching for Fast Image-to-Image Translationcited by 36
  6. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  7. 2024IllumiNeRF: 3D Relighting Without Inverse Renderingcited by 29
  8. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  9. 2025Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Datasetcited by 28
  10. 2024ScribbleLight: Single Image Indoor Relighting with Scribblescited by 24
  11. 2024GenLit: Reformulating Single-Image Relighting as Video Generationcited by 20
  12. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 19
38 more
  1. 2024Latent Intrinsics Emerge from Training to Relightcited by 17
  2. 2025IntrinsicEdit: Precise generative image manipulation in intrinsic spacecited by 17
  3. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
  4. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Renderingcited by 16
  5. 2025TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfercited by 11
  6. 2024Boosting 3D Object Generation through PBR Materialscited by 10
  7. 2026LuxRemix: Lighting Decomposition and Remixing for Indoor Scenescited by 9
  8. 2024Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theorycited by 8
  9. 2025RelightMaster: Precise Video Relighting with Multi-plane Light Imagescited by 8
  10. 2024Birth and Death of a Rosecited by 7
  11. 2025IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generationcited by 7
  12. 2025PractiLight: Practical Light Control Using Foundational Diffusion Modelscited by 7
  13. 2025MatSwap: Light-aware material transfers in imagescited by 6
  14. 2025Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Modelscited by 5
  15. 2026PI-Light: Physics-Inspired Diffusion for Full-Image Relightingcited by 5
  16. 2025UniLight: A Unified Representation for Lightingcited by 4
  17. 2024LumiSculpt: Enabling Consistent Portrait Lighting in Video Generationcited by 3
  18. 2025Refracting Reality: Generating Images with Realistic Transparent Objectscited by 2
  19. 2025GimbalDiffusion: Gravity-Aware Camera Control for Video Generationcited by 2
  20. 2026Light4D: Training-Free Extreme Viewpoint 4D Video Relightingcited by 2
  21. 2024MLI-NeRF: Multi-Light Intrinsic-Aware Neural Radiance Fieldscited by 1
  22. 2025Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Groundingcited by 1
  23. 2025Yesnt: Are Diffusion Relighting Models Ready for Capture Stage Compositing? A Hybrid Alternative to Bridge the Gapcited by 1
  24. 2025LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Modelscited by 1
  25. 2026LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulationcited by 1
  26. 2026TokenLight: Precise Lighting Control in Images using Attribute Tokenscited by 1
  27. 2026F-RNG: Feed-Forward Relightable Neural Gaussianscited by 1
  28. 2026HorizonRelight: Relighting Long-horizon Videos Consistently via Diffusion Transformerscited by 1
  29. 2026Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relightingcited by 1
  30. 2026RGBX-Next: Towards Realistic Generative Rendering from G-Bufferscited by 1
  31. 2024Learning Relighting and Intrinsic Decomposition in Neural Radiance Fieldscited by 0
  32. 2025Make Your MoVe: Make Your 3D Contents by Adapting Multi-View Diffusion Models to External Editingcited by 0
  33. 2026RelightAnyone: A Generalized Relightable 3D Gaussian Head Modelcited by 0
  34. 2026SyncLight: Single-Edit Multi-View Relightingcited by 0
  35. 2026Relighting as a Probe of Visual Priors via Augmented Latent Intrinsicscited by 0
  36. 2026Joint Shadow Generation and Relighting via Light-Geometry Interaction Mapscited by 0
  37. 2026VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusioncited by 0
  38. 2026CamLit: Unified Video Diffusion with Explicit Camera and Lighting Controlcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Illumination Control in Diffusion Models
  2. 2026LightCtrl: Training-free Controllable Video Relighting
  3. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation
  4. 2026LightMover: Generative Light Movement with Color and Intensity Controls
  5. 2024SpotLight: Shadow-Guided Object Relighting via Diffusion
  6. 2025TransLight: Image-Guided Customized Lighting Control with Generative Decoupling
  7. 2026Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction
  8. 2026Learning Latent Proxies for Controllable Single-Image Relighting