# Learning Relighting and Intrinsic Decomposition in Neural Radiance Fields

arXiv: https://arxiv.org/abs/2406.11077 · HTML: https://mlsanity.com/paper/2406.11077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering](https://mlsanity.com/paper/1808.08601/) (2018)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [Unsupervised Learning for Intrinsic Image Decomposition from a Single Image](https://mlsanity.com/paper/1911.09930/) (2019)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [A Dataset of Multi-Illumination Images in the Wild](https://mlsanity.com/paper/1910.08131/) (2019)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects](https://mlsanity.com/paper/2304.10448/) (2023)
- [Relighting Neural Radiance Fields with Shadow and Highlight Hints](https://mlsanity.com/paper/2308.13404/) (2023)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition](https://mlsanity.com/paper/2203.16670/) (2022)
- [NTIRE 2021 Depth Guided Image Relighting Challenge](https://mlsanity.com/paper/2104.13365/) (2021)
- [Unsupervised Deep Single-Image Intrinsic Decomposition using Illumination-Varying Image Sequences](https://mlsanity.com/paper/1803.00805/) (2018)
- [SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes](https://mlsanity.com/paper/2310.13030/) (2023)
- [AIM 2020: Scene Relighting and Illumination Estimation Challenge](https://mlsanity.com/paper/2009.12798/) (2020)
- [ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision](https://mlsanity.com/paper/2211.14086/) (2022)
- [Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis](https://mlsanity.com/paper/2308.02840/) (2023)
- [WDRN : A Wavelet Decomposed RelightNet for Image Relighting](https://mlsanity.com/paper/2009.06678/) (2020)
- [Deep Relighting Networks for Image Light Source Manipulation](https://mlsanity.com/paper/2008.08298/) (2020)

## Abstract (arXiv)

The task of extracting intrinsic components, such as reflectance and shading, from neural radiance fields is of growing interest. However, current methods largely focus on synthetic scenes and isolated objects, overlooking the complexities of real scenes with backgrounds. To address this gap, our research introduces a method that combines relighting with intrinsic decomposition. By leveraging light variations in scenes to generate pseudo labels, our method provides guidance for intrinsic decomposition without requiring ground truth data. Our method, grounded in physical constraints, ensures robustness across diverse scene types and reduces the reliance on pre-trained models or hand-crafted priors. We validate our method on both synthetic and real-world datasets, achieving convincing results. Furthermore, the applicability of our method to image editing tasks demonstrates promising outcomes.