# NeILF: Neural Incident Light Field for Physically-based Material Estimation

arXiv: https://arxiv.org/abs/2203.07182 · HTML: https://mlsanity.com/paper/2203.07182/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing](https://mlsanity.com/paper/1911.13225/) (2019)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [Learning to Infer Implicit Surfaces without 3D Supervision](https://mlsanity.com/paper/1911.00767/) (2019)
- [Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images](https://mlsanity.com/paper/2007.09892/) (2020)
- [Visibility-aware Multi-view Stereo Network](https://mlsanity.com/paper/2008.07928/) (2020)
- [Learning Signed Distance Field for Multi-view Surface Reconstruction](https://mlsanity.com/paper/2108.09964/) (2021)
- [Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images](https://mlsanity.com/paper/2003.12642/) (2020)
- [Inverse Path Tracing for Joint Material and Lighting Estimation](https://mlsanity.com/paper/1903.07145/) (2019)
- [DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer](https://mlsanity.com/paper/2111.00140/) (2021)
- [Seeing the World in a Bag of Chips](https://mlsanity.com/paper/2001.04642/) (2020)

## Built on by (most cited first, 30 of 105)

- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing](https://mlsanity.com/paper/2311.16043/) (2023)
- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes](https://mlsanity.com/paper/2304.03266/) (2023)
- [GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorization](https://mlsanity.com/paper/2312.05133/) (2023)
- [NeILF++: Inter-Reflectable Light Fields for Geometry and Material Estimation](https://mlsanity.com/paper/2303.17147/) (2023)
- [IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination](https://mlsanity.com/paper/2404.11593/) (2024)
- [DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models](https://mlsanity.com/paper/2405.17176/) (2024)
- [GaussianHair: Hair Modeling and Rendering with Light-aware Gaussians](https://mlsanity.com/paper/2402.10483/) (2024)
- [GI-GS: Global Illumination Decomposition on Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2410.02619/) (2024)
- [A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis](https://mlsanity.com/paper/2409.08947/) (2024)
- [DifFRelight: Diffusion-Based Facial Performance Relighting](https://mlsanity.com/paper/2410.08188/) (2024)
- [SpecNeRF: Gaussian Directional Encoding for Specular Reflections](https://mlsanity.com/paper/2312.13102/) (2023)
- [Ref-GS: Directional Factorization for 2D Gaussian Splatting](https://mlsanity.com/paper/2412.00905/) (2024)
- [NeISF: Neural Incident Stokes Field for Geometry and Material Estimation](https://mlsanity.com/paper/2311.13187/) (2023)
- [Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes](https://mlsanity.com/paper/2211.10206/) (2022)
- [Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation](https://mlsanity.com/paper/2304.05669/) (2023)
- [SVG-IR: Spatially-Varying Gaussian Splatting for Inverse Rendering](https://mlsanity.com/paper/2504.06815/) (2025)
- [Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh](https://mlsanity.com/paper/2405.17811/) (2024)
- [Dynamic Mesh-Aware Radiance Fields](https://mlsanity.com/paper/2309.04581/) (2023)
- [MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections](https://mlsanity.com/paper/2405.11921/) (2024)
- [IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images](https://mlsanity.com/paper/2401.12977/) (2024)
- [SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes](https://mlsanity.com/paper/2310.13030/) (2023)
- [SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild](https://mlsanity.com/paper/2401.10171/) (2024)
- [PRTGS: Precomputed Radiance Transfer of Gaussian Splats for Real-Time High-Quality Relighting](https://mlsanity.com/paper/2408.03538/) (2024)
- [iVR-GS: Inverse Volume Rendering for Explorable Visualization via Editable 3D Gaussian Splatting](https://mlsanity.com/paper/2504.17954/) (2025)
- [ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision](https://mlsanity.com/paper/2211.14086/) (2022)
- [Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields](https://mlsanity.com/paper/2403.16224/) (2024)
- [VDN-NeRF: Resolving Shape-Radiance Ambiguity via View-Dependence Normalization](https://mlsanity.com/paper/2303.17968/) (2023)
- [Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Field](https://mlsanity.com/paper/2304.03526/) (2023)

## Abstract (arXiv)

We present a differentiable rendering framework for material and lighting estimation from multi-view images and a reconstructed geometry. In the framework, we represent scene lightings as the Neural Incident Light Field (NeILF) and material properties as the surface BRDF modelled by multi-layer perceptrons. Compared with recent approaches that approximate scene lightings as the 2D environment map, NeILF is a fully 5D light field that is capable of modelling illuminations of any static scenes. In addition, occlusions and indirect lights can be handled naturally by the NeILF representation without requiring multiple bounces of ray tracing, making it possible to estimate material properties even for scenes with complex lightings and geometries. We also propose a smoothness regularization and a Lambertian assumption to reduce the material-lighting ambiguity during the optimization. Our method strictly follows the physically-based rendering equation, and jointly optimizes material and lighting through the differentiable rendering process. We have intensively evaluated the proposed method on our in-house synthetic dataset, the DTU MVS dataset, and real-world BlendedMVS scenes. Our method is able to outperform previous methods by a significant margin in terms of novel view rendering quality, setting a new state-of-the-art for image-based material and lighting estimation.