# Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination

arXiv: https://arxiv.org/abs/2003.08367 · HTML: https://mlsanity.com/paper/2003.08367/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 30 of 63)

- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Mixed Neural Voxels for Fast Multi-view Video Synthesis](https://mlsanity.com/paper/2212.00190/) (2022)
- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [NeILF++: Inter-Reflectable Light Fields for Geometry and Material Estimation](https://mlsanity.com/paper/2303.17147/) (2023)
- [InstantSplat: Sparse-view Gaussian Splatting in Seconds](https://mlsanity.com/paper/2403.20309/) (2024)
- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes](https://mlsanity.com/paper/2206.08423/) (2022)
- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations](https://mlsanity.com/paper/2207.01164/) (2022)
- [Neural Deformable Voxel Grid for Fast Optimization of Dynamic View Synthesis](https://mlsanity.com/paper/2206.07698/) (2022)
- [Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels](https://mlsanity.com/paper/2405.16822/) (2024)
- [Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion](https://mlsanity.com/paper/2208.09480/) (2022)
- [Guided Co-Modulated GAN for 360deg Field of View Extrapolation](https://mlsanity.com/paper/2204.07286/) (2022)
- [EverLight: Indoor-Outdoor Editable HDR Lighting Estimation](https://mlsanity.com/paper/2304.13207/) (2023)
- [SpecNeRF: Gaussian Directional Encoding for Specular Reflections](https://mlsanity.com/paper/2312.13102/) (2023)
- [GMLight: Lighting Estimation via Geometric Distribution Approximation](https://mlsanity.com/paper/2102.10244/) (2021)
- [Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering](https://mlsanity.com/paper/2408.09702/) (2024)
- [Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation](https://mlsanity.com/paper/2304.05669/) (2023)
- [ORCa: Glossy Objects as Radiance Field Cameras](https://mlsanity.com/paper/2212.04531/) (2022)
- [IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images](https://mlsanity.com/paper/2401.12977/) (2024)
- [Sparse Needlets for Lighting Estimation with Spherical Transport Loss](https://mlsanity.com/paper/2106.13090/) (2021)
- [Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluation](https://mlsanity.com/paper/2306.15662/) (2023)
- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields](https://mlsanity.com/paper/2504.20026/) (2025)
- [Accidental Light Probes](https://mlsanity.com/paper/2301.05211/) (2023)
- [Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation](https://mlsanity.com/paper/2412.15211/) (2024)
- [Editable Indoor Lighting Estimation](https://mlsanity.com/paper/2211.03928/) (2022)
- [MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimation](https://mlsanity.com/paper/2303.12368/) (2023)

## Abstract (arXiv)

We present a deep learning solution for estimating the incident illumination at any 3D location within a scene from an input narrow-baseline stereo image pair. Previous approaches for predicting global illumination from images either predict just a single illumination for the entire scene, or separately estimate the illumination at each 3D location without enforcing that the predictions are consistent with the same 3D scene. Instead, we propose a deep learning model that estimates a 3D volumetric RGBA model of a scene, including content outside the observed field of view, and then uses standard volume rendering to estimate the incident illumination at any 3D location within that volume. Our model is trained without any ground truth 3D data and only requires a held-out perspective view near the input stereo pair and a spherical panorama taken within each scene as supervision, as opposed to prior methods for spatially-varying lighting estimation, which require ground truth scene geometry for training. We demonstrate that our method can predict consistent spatially-varying lighting that is convincing enough to plausibly relight and insert highly specular virtual objects into real images.