# S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint

arXiv: https://arxiv.org/abs/2210.08936 · HTML: https://mlsanity.com/paper/2210.08936/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 88 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 30 of 46)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022)
- [ShaRF: Shape-conditioned Radiance Fields from a Single View](https://mlsanity.com/paper/2102.08860/) (2021)
- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [Portrait Neural Radiance Fields from a Single Image](https://mlsanity.com/paper/2012.05903/) (2020)
- [Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials](https://mlsanity.com/paper/2001.06659/) (2020)

## Built on by (most cited first, 30 of 30)

- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [Forward Flow for Novel View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2309.17390/) (2023)
- [NeISF: Neural Incident Stokes Field for Geometry and Material Estimation](https://mlsanity.com/paper/2311.13187/) (2023)
- [UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video](https://mlsanity.com/paper/2306.09349/) (2023)
- [Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery](https://mlsanity.com/paper/2304.03384/) (2023)
- [Looking Through the Glass: Neural Surface Reconstruction Against High Specular Reflections](https://mlsanity.com/paper/2304.08706/) (2023)
- [ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision](https://mlsanity.com/paper/2211.14086/) (2022)
- [Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects](https://mlsanity.com/paper/2305.17929/) (2023)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [Deep Learning Methods for Calibrated Photometric Stereo and Beyond](https://mlsanity.com/paper/2212.08414/) (2022)
- [NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination](https://mlsanity.com/paper/2304.08757/) (2023)
- [PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar](https://mlsanity.com/paper/2312.14239/) (2023)
- [SUNDIAL: 3D Satellite Understanding through Direct, Ambient, and Complex Lighting Decomposition](https://mlsanity.com/paper/2312.16215/) (2023)
- [NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering](https://mlsanity.com/paper/2402.04829/) (2024)
- [OpenOcc: Open Vocabulary 3D Scene Reconstruction via Occupancy Representation](https://mlsanity.com/paper/2403.11796/) (2024)
- [DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering](https://mlsanity.com/paper/2303.15101/) (2023)
- [NPLMV-PS: Neural Point-Light Multi-View Photometric Stereo](https://mlsanity.com/paper/2405.12057/) (2024)
- [A Neural Height-Map Approach for the Binocular Photometric Stereo Problem](https://mlsanity.com/paper/2311.05958/) (2023)
- [G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images](https://mlsanity.com/paper/2404.07474/) (2024)
- [Surface Normal Clustering for Implicit Representation of Manhattan Scenes](https://mlsanity.com/paper/2212.01331/) (2022)
- [ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images](https://mlsanity.com/paper/2404.15707/) (2024)
- [PIR: Photometric Inverse Rendering with Shading Cues Modeling and Surface Reflectance Regularization](https://mlsanity.com/paper/2408.06828/) (2024)
- [BRDF-NeRF: Neural Radiance Fields with Optical Satellite Images and BRDF Modelling](https://mlsanity.com/paper/2409.12014/) (2024)
- [LUCES-MV: A Multi-View Dataset for Near-Field Point Light Source Photometric Stereo](https://mlsanity.com/paper/2412.16737/) (2024)
- [Relative Illumination Fields: Learning Medium and Light Independent Underwater Scenes](https://mlsanity.com/paper/2504.10024/) (2025)
- [GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies](https://mlsanity.com/paper/2511.00998/) (2025)
- [Mixed Reality Interface for Digital Twin of Plant Factory](https://mlsanity.com/paper/2211.00597/) (2022)
- [Shape from Shading for Robotic Manipulation](https://mlsanity.com/paper/2304.11824/) (2023)
- [NePF: Neural Photon Field for Single-Stage Inverse Rendering](https://mlsanity.com/paper/2311.11555/) (2023)
- [RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing](https://mlsanity.com/paper/2603.27948/) (2026)

## Abstract (arXiv)

In this paper, we address the "dual problem" of multi-view scene reconstruction in which we utilize single-view images captured under different point lights to learn a neural scene representation. Different from existing single-view methods which can only recover a 2.5D scene representation (i.e., a normal / depth map for the visible surface), our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene. Instead of relying on multi-view photo-consistency, our method exploits two information-rich monocular cues, namely shading and shadow, to infer scene geometry. Experiments on multiple challenging datasets show that our method is capable of recovering 3D geometry, including both visible and invisible parts, of a scene from single-view images. Thanks to the neural reflectance field representation, our method is robust to depth discontinuities. It supports applications like novel-view synthesis and relighting. Our code and model can be found at https://ywq.github.io/s3nerf.