# Neural Light Transport for Relighting and View Synthesis

arXiv: https://arxiv.org/abs/2008.03806 · HTML: https://mlsanity.com/paper/2008.03806/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2020.

## Built on by (most cited first, 30 of 73)

- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [BACON: Band-limited Coordinate Networks for Multiscale Scene Representation](https://mlsanity.com/paper/2112.04645/) (2021)
- [NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild](https://mlsanity.com/paper/2110.07604/) (2021)
- [NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination](https://mlsanity.com/paper/2106.01970/) (2021)
- [Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2312.03704/) (2023)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Neural Lumigraph Rendering](https://mlsanity.com/paper/2103.11571/) (2021)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [ACORN: Adaptive Coordinate Networks for Neural Scene Representation](https://mlsanity.com/paper/2105.02788/) (2021)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data](https://mlsanity.com/paper/2206.06577/) (2022)
- [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021)
- [Neural Video Portrait Relighting in Real-time via Consistency Modeling](https://mlsanity.com/paper/2104.00484/) (2021)
- [Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN](https://mlsanity.com/paper/2109.06166/) (2021)
- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects](https://mlsanity.com/paper/2304.10448/) (2023)
- [Relighting Neural Radiance Fields with Shadow and Highlight Hints](https://mlsanity.com/paper/2308.13404/) (2023)
- [Relighting4D: Neural Relightable Human from Videos](https://mlsanity.com/paper/2207.07104/) (2022)
- [Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination](https://mlsanity.com/paper/2207.13607/) (2022)
- [NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting](https://mlsanity.com/paper/2107.12351/) (2021)
- [NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination](https://mlsanity.com/paper/2303.16617/) (2023)
- [LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting](https://mlsanity.com/paper/2412.00177/) (2024)
- [A Dataset of Relighted 3D Interacting Hands](https://mlsanity.com/paper/2310.17768/) (2023)
- [DifFRelight: Diffusion-Based Facial Performance Relighting](https://mlsanity.com/paper/2410.08188/) (2024)
- [NeMF: Inverse Volume Rendering with Neural Microflake Field](https://mlsanity.com/paper/2304.00782/) (2023)
- [NeuLF: Efficient Novel View Synthesis with Neural 4D Light Field](https://mlsanity.com/paper/2105.07112/) (2021)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [Single-image Full-body Human Relighting](https://mlsanity.com/paper/2107.07259/) (2021)

## Abstract (arXiv)

The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination, while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse, previously seen observations. Qualitative and quantitative experiments demonstrate that our neural LT (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without separate treatment for both problems that prior work requires.