# Relighting Neural Radiance Fields with Shadow and Highlight Hints

arXiv: https://arxiv.org/abs/2308.13404 · HTML: https://mlsanity.com/paper/2308.13404/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 67 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination](https://mlsanity.com/paper/2106.01970/) (2021)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [NeRF for Outdoor Scene Relighting](https://mlsanity.com/paper/2112.05140/) (2021)
- [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022)
- [Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images](https://mlsanity.com/paper/2007.09892/) (2020)
- [SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections](https://mlsanity.com/paper/2205.15768/) (2022)
- [Neural Light Transport for Relighting and View Synthesis](https://mlsanity.com/paper/2008.03806/) (2020)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer](https://mlsanity.com/paper/2111.00140/) (2021)
- [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021)
- [Deep Shading: Convolutional Neural Networks for Screen-Space Shading](https://mlsanity.com/paper/1603.06078/) (2016)
- [Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination](https://mlsanity.com/paper/2207.13607/) (2022)

## Built on by (most cited first, 30 of 42)

- [GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorization](https://mlsanity.com/paper/2312.05133/) (2023)
- [GS^3: Efficient Relighting with Triple Gaussian Splatting](https://mlsanity.com/paper/2410.11419/) (2024)
- [A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis](https://mlsanity.com/paper/2409.08947/) (2024)
- [DifFRelight: Diffusion-Based Facial Performance Relighting](https://mlsanity.com/paper/2410.08188/) (2024)
- [SpecNeRF: Gaussian Directional Encoding for Specular Reflections](https://mlsanity.com/paper/2312.13102/) (2023)
- [Rip-NeRF: Anti-aliasing Radiance Fields with Ripmap-Encoded Platonic Solids](https://mlsanity.com/paper/2405.02386/) (2024)
- [LumiGauss: Relightable Gaussian Splatting in the Wild](https://mlsanity.com/paper/2408.04474/) (2024)
- [RNG: Relightable Neural Gaussians](https://mlsanity.com/paper/2409.19702/) (2024)
- [Localized Gaussian Splatting Editing with Contextual Awareness](https://mlsanity.com/paper/2408.00083/) (2024)
- [GlossyGS: Inverse Rendering of Glossy Objects with 3D Gaussian Splatting](https://mlsanity.com/paper/2410.13349/) (2024)
- [GaRe: Relightable 3D Gaussian Splatting for Outdoor Scenes from Unconstrained Photo Collections](https://mlsanity.com/paper/2507.20512/) (2025)
- [BiGS: Bidirectional Gaussian Primitives for Relightable 3D Gaussian Splatting](https://mlsanity.com/paper/2408.13370/) (2024)
- [ARM: Appearance Reconstruction Model for Relightable 3D Generation](https://mlsanity.com/paper/2411.10825/) (2024)
- [ROGR: Relightable 3D Objects using Generative Relighting](https://mlsanity.com/paper/2510.03163/) (2025)
- [Artist-Friendly Relightable and Animatable Neural Heads](https://mlsanity.com/paper/2312.03420/) (2023)
- [Differentiable Inverse Rendering with Interpretable Basis BRDFs](https://mlsanity.com/paper/2411.17994/) (2024)
- [Relighting Scenes with Object Insertions in Neural Radiance Fields](https://mlsanity.com/paper/2406.14806/) (2024)
- [BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video](https://mlsanity.com/paper/2502.08297/) (2025)
- [OLATverse: A Large-scale Real-world Object Dataset with Precise Lighting Control](https://mlsanity.com/paper/2511.02483/) (2025)
- [NeAR: Coupled Neural Asset-Renderer Stack](https://mlsanity.com/paper/2511.18600/) (2025)
- [MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting](https://mlsanity.com/paper/2405.20791/) (2024)
- [PRTGaussian: Efficient Relighting Using 3D Gaussians with Precomputed Radiance Transfer](https://mlsanity.com/paper/2408.05631/) (2024)
- [A Real-world Display Inverse Rendering Dataset](https://mlsanity.com/paper/2508.14411/) (2025)
- [Differentiable Light Transport with Gaussian Surfels via Adapted Radiosity for Efficient Relighting and Geometry Reconstruction](https://mlsanity.com/paper/2509.18497/) (2025)
- [RNA: Relightable Neural Assets](https://mlsanity.com/paper/2312.09398/) (2023)
- [ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images](https://mlsanity.com/paper/2404.15707/) (2024)
- [PIR: Photometric Inverse Rendering with Shading Cues Modeling and Surface Reflectance Regularization](https://mlsanity.com/paper/2408.06828/) (2024)
- [On Neural BRDFs: A Thorough Comparison of State-of-the-Art Approaches](https://mlsanity.com/paper/2502.15480/) (2025)
- [Real-time Global Illumination for Dynamic 3D Gaussian Scenes](https://mlsanity.com/paper/2503.17897/) (2025)
- [Relative Illumination Fields: Learning Medium and Light Independent Underwater Scenes](https://mlsanity.com/paper/2504.10024/) (2025)

## Abstract (arXiv)

This paper presents a novel neural implicit radiance representation for free viewpoint relighting from a small set of unstructured photographs of an object lit by a moving point light source different from the view position. We express the shape as a signed distance function modeled by a multi layer perceptron. In contrast to prior relightable implicit neural representations, we do not disentangle the different reflectance components, but model both the local and global reflectance at each point by a second multi layer perceptron that, in addition, to density features, the current position, the normal (from the signed distace function), view direction, and light position, also takes shadow and highlight hints to aid the network in modeling the corresponding high frequency light transport effects. These hints are provided as a suggestion, and we leave it up to the network to decide how to incorporate these in the final relit result. We demonstrate and validate our neural implicit representation on synthetic and real scenes exhibiting a wide variety of shapes, material properties, and global illumination light transport.