# Object-based Illumination Estimation with Rendering-aware Neural Networks

arXiv: https://arxiv.org/abs/2008.02514 · HTML: https://mlsanity.com/paper/2008.02514/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2020.

## Built on by (most cited first, 19 of 19)

- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination](https://mlsanity.com/paper/2106.01970/) (2021)
- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021)
- [NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination](https://mlsanity.com/paper/2303.16617/) (2023)
- [Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion](https://mlsanity.com/paper/2208.09480/) (2022)
- [NeMF: Inverse Volume Rendering with Neural Microflake Field](https://mlsanity.com/paper/2304.00782/) (2023)
- [IntrinsicAvatar: Physically Based Inverse Rendering of Dynamic Humans from Monocular Videos via Explicit Ray Tracing](https://mlsanity.com/paper/2312.05210/) (2023)
- [LightSim: Neural Lighting Simulation for Urban Scenes](https://mlsanity.com/paper/2312.06654/) (2023)
- [Accidental Light Probes](https://mlsanity.com/paper/2301.05211/) (2023)
- [Spatiotemporally Consistent HDR Indoor Lighting Estimation](https://mlsanity.com/paper/2305.04374/) (2023)
- [Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2407.15212/) (2024)
- [Diffusion Reflectance Map: Single-Image Stochastic Inverse Rendering of Illumination and Reflectance](https://mlsanity.com/paper/2312.04529/) (2023)
- [Differentiable Point-based Inverse Rendering](https://mlsanity.com/paper/2312.02480/) (2023)
- [Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction](https://mlsanity.com/paper/2202.06300/) (2022)
- [Differentiable Inverse Rendering with Interpretable Basis BRDFs](https://mlsanity.com/paper/2411.17994/) (2024)
- [A Real-world Display Inverse Rendering Dataset](https://mlsanity.com/paper/2508.14411/) (2025)
- [Sun Off, Lights On: Photorealistic Monocular Nighttime Simulation for Robust Semantic Perception](https://mlsanity.com/paper/2407.20336/) (2024)
- [Under One Sun: Multi-Object Generative Perception of Materials and Illumination](https://mlsanity.com/paper/2603.19226/) (2026)

## Abstract (arXiv)

We present a scheme for fast environment light estimation from the RGBD appearance of individual objects and their local image areas. Conventional inverse rendering is too computationally demanding for real-time applications, and the performance of purely learning-based techniques may be limited by the meager input data available from individual objects. To address these issues, we propose an approach that takes advantage of physical principles from inverse rendering to constrain the solution, while also utilizing neural networks to expedite the more computationally expensive portions of its processing, to increase robustness to noisy input data as well as to improve temporal and spatial stability. This results in a rendering-aware system that estimates the local illumination distribution at an object with high accuracy and in real time. With the estimated lighting, virtual objects can be rendered in AR scenarios with shading that is consistent to the real scene, leading to improved realism.