mlsanityData to 7 Oct 2026

DiffusionLight: Light Probes for Free by Painting a Chrome Ball

arXiv 2312.09168 ↗First posted 14 Dec 2023cs.CV cs.GR cs.LGcited by 63

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present a simple yet effective technique to estimate lighting in a single input image. Current techniques rely heavily on HDR panorama datasets to train neural networks to regress an input with limited field-of-view to a full environment map. However, these approaches often struggle with real-world, uncontrolled settings due to the limited diversity and size of their datasets. To address this problem, we leverage diffusion models trained on billions of standard images to render a chrome ball into the input image. Despite its simplicity, this task remains challenging: the diffusion models often insert incorrect or inconsistent objects and cannot readily generate images in HDR format. Our research uncovers a surprising relationship between the appearance of chrome balls and the initial diffusion noise map, which we utilize to consistently generate high-quality chrome balls. We further fine-tune an LDR diffusion model (Stable Diffusion XL) with LoRA, enabling it to perform exposure bracketing for HDR light estimation. Our method produces convincing light estimates across diverse settings and demonstrates superior generalization to in-the-wild scenarios.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 53 of its 68 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

53 of 68 references.

References · 53

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2021Vision Transformers for Dense Predictioncited by 1,789
  12. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
38 more
  1. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  4. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 826
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  6. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  7. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  8. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  9. 2022Blended Latent Diffusioncited by 430
  10. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  11. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  12. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  13. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  14. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  15. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  16. 2017Learning to Predict Indoor Illumination from a Single Imagecited by 160
  17. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  18. 2018ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Contentcited by 89
  19. 2019Deep Parametric Indoor Lighting Estimationcited by 82
  20. 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 72
  21. 2019Rendering Synthetic Objects into Legacy Photographscited by 72
  22. 2019Deep Sky Modeling for Single Image Outdoor Lighting Estimationcited by 66
  23. 2019DeepLight: Learning Illumination for Unconstrained Mobile Mixed Realitycited by 63
  24. 2022StyleLight: HDR Panorama Generation for Lighting Estimation and Editingcited by 59
  25. 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
  26. 2016Deep Outdoor Illumination Estimationcited by 53
  27. 2020EMLight: Lighting Estimation via Spherical Distribution Approximationcited by 47
  28. 2022Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creationcited by 47
  29. 2020HDR Environment Map Estimation for Real-Time Augmented Realitycited by 46
  30. 2023360-Degree Panorama Generation from Few Unregistered NFoV Imagescited by 42
  31. 2019Automatic Scene Inference for 3D Object Compositingcited by 36
  32. 2022Guided Co-Modulated GAN for 360deg Field of View Extrapolationcited by 32
  33. 2023EverLight: Indoor-Outdoor Editable HDR Lighting Estimationcited by 32
  34. 2021GMLight: Lighting Estimation via Geometric Distribution Approximationcited by 28
  35. 2020Seeing the World in a Bag of Chipscited by 25
  36. 2022Text2Light: Zero-Shot Text-Driven HDR Panorama Generationcited by 24
  37. 2018Learning to Estimate Indoor Lighting from 3D Objectscited by 17
  38. 2023Accidental Light Probescited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 33; 2026: 17.

63 papers.

Sort:

By area:

  1. 2024Neural Gaffer: Relighting Any Object via Diffusioncited by 91
  2. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 65
  3. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  4. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  5. 2024LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relightingcited by 37
  6. 2024IllumiNeRF: 3D Relighting Without Inverse Renderingcited by 29
  7. 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Renderingcited by 27
  8. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  9. 20243DSceneEditor: Controllable 3D Scene Editing with Gaussian Splattingcited by 20
  10. 2024GenLit: Reformulating Single-Image Relighting as Video Generationcited by 20
  11. 2024ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusioncited by 16
  12. 2025Towards Physically Executable 3D Gaussian for Embodied Navigationcited by 16
38 more
  1. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  2. 2024Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variationcited by 13
  3. 2025A Recipe for Generating 3D Worlds From a Single Imagecited by 13
  4. 2025Orientation Matters: Making 3D Generative Models Orientation-Alignedcited by 13
  5. 2025GaSLight: Gaussian Splats for Spatially-Varying Lighting in HDRcited by 11
  6. 2025DiffusionLight-Turbo: Accelerated Light Probes for Free via Single-Pass Chrome Ball Inpaintingcited by 11
  7. 2025Light-X: Generative 4D Video Rendering with Camera and Illumination Controlcited by 11
  8. 2025Materialist: Physically Based Editing Using Single-Image Inverse Renderingcited by 10
  9. 2025GCC: Generative Color Constancy via Diffusing a Color Checkercited by 8
  10. 2024RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Imagecited by 7
  11. 2024UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulationcited by 7
  12. 20253D Object Manipulation in a Single Image using Generative Modelscited by 7
  13. 2025IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generationcited by 6
  14. 2025MVInverse: Feed-forward Multi-view Inverse Rendering in Secondscited by 5
  15. 2025InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modelingcited by 4
  16. 2025V-RGBX: Video Editing with Accurate Controls over Intrinsic Propertiescited by 4
  17. 2024CleAR: Robust Context-Guided Generative Lighting Estimation for Mobile Augmented Realitycited by 3
  18. 2025ePBR: Extended PBR Materials in Image Synthesiscited by 3
  19. 2025MADrive: Memory-Augmented Driving Scene Modelingcited by 3
  20. 2025UniLight: A Unified Representation for Lightingcited by 3
  21. 2025Diffusion Models are Secretly Zero-Shot 3DGS Harmonizerscited by 2
  22. 2025Twinner: Shining Light on Digital Twins in a Few Snapscited by 2
  23. 2025TransLight: Image-Guided Customized Lighting Control with Generative Decouplingcited by 2
  24. 2025Refracting Reality: Generating Images with Realistic Transparent Objectscited by 2
  25. 2025Channel-wise Noise Scheduled Diffusion for Inverse Rendering in Indoor Scenescited by 1
  26. 2025UAVTwin: Neural Digital Twins for UAVs using Gaussian Splattingcited by 1
  27. 2025ComGS: Efficient 3D Object-Scene Composition via Surface Octahedral Probescited by 1
  28. 2025Lighting in Motion: Spatiotemporal HDR Lighting Estimationcited by 1
  29. 2026VENI: Variational Encoder for Natural Illuminationcited by 1
  30. 2026ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illuminationcited by 1
  31. 2026Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relightingcited by 1
  32. 2024PanDORA: Casual HDR Radiance Acquisition of Indoor Scenes for Image-based Lightingcited by 0
  33. 2025AR as an Evaluation Playground: Bridging Metrics and Visual Perception of Computer Vision Modelscited by 0
  34. 2025Improving the color accuracy of lighting estimation modelscited by 0
  35. 2025Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Skycited by 0
  36. 2025Cast and Attached Shadow Detection via Iterative Light and Geometry Reasoningcited by 0
  37. 2025Towards Physically-Based Sky-Modeling For Image Based Lightingcited by 0
  38. 2026HDR Reconstruction Boosting with Training-Free and Exposure-Consistent Diffusioncited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LightLab: Controlling Light Sources in Images with Diffusion Models
  2. 2026Learning Illumination Control in Diffusion Models
  3. 2026PI-Light: Physics-Inspired Diffusion for Full-Image Relighting
  4. 2024Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion
  5. 2025Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors
  6. 2025PractiLight: Practical Light Control Using Foundational Diffusion Models
  7. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis
  8. 2024MatFusion: A Generative Diffusion Model for SVBRDF Capture