# PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo

arXiv: https://arxiv.org/abs/2207.11406 · HTML: https://mlsanity.com/paper/2207.11406/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 67 references that are in the map.
- First posted on arXiv 23 Jul 2022.

## Builds on (most cited first, 30 of 33)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [Neural Reflectance Fields for Appearance Acquisition](https://mlsanity.com/paper/2008.03824/) (2020)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials](https://mlsanity.com/paper/2001.06659/) (2020)
- [PS-FCN: A Flexible Learning Framework for Photometric Stereo](https://mlsanity.com/paper/1807.08696/) (2018)
- [Self-calibrating Deep Photometric Stereo Networks](https://mlsanity.com/paper/1903.07366/) (2019)
- [CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces](https://mlsanity.com/paper/1808.10093/) (2018)
- [Neural Inverse Rendering for General Reflectance Photometric Stereo](https://mlsanity.com/paper/1802.10328/) (2018)
- [Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces](https://mlsanity.com/paper/2012.06777/) (2020)
- [Uncertainty-Aware Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2202.13071/) (2022)
- [Neural Reflectance for Shape Recovery with Shadow Handling](https://mlsanity.com/paper/2203.12909/) (2022)
- [Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2110.05594/) (2021)
- [SPLINE-Net: Sparse Photometric Stereo through Lighting Interpolation and Normal Estimation Networks](https://mlsanity.com/paper/1905.04088/) (2019)
- [Deep Photometric Stereo for Non-Lambertian Surfaces](https://mlsanity.com/paper/2007.13145/) (2020)

## Built on by (most cited first, 30 of 69)

- [SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://mlsanity.com/paper/2211.11738/) (2022)
- [NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images](https://mlsanity.com/paper/2305.17398/) (2023)
- [BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields](https://mlsanity.com/paper/2211.12853/) (2022)
- [Fully Explicit Dynamic Gaussian Splatting](https://mlsanity.com/paper/2410.15629/) (2024)
- [DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models](https://mlsanity.com/paper/2405.17176/) (2024)
- [Relighting Neural Radiance Fields with Shadow and Highlight Hints](https://mlsanity.com/paper/2308.13404/) (2023)
- [Efficient Large-scale Scene Representation with a Hybrid of High-resolution Grid and Plane Features](https://mlsanity.com/paper/2303.03003/) (2023)
- [S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint](https://mlsanity.com/paper/2210.08936/) (2022)
- [Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes](https://mlsanity.com/paper/2211.10206/) (2022)
- [MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference](https://mlsanity.com/paper/2510.11387/) (2025)
- [SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes](https://mlsanity.com/paper/2310.13030/) (2023)
- [MVPSNet: Fast Generalizable Multi-view Photometric Stereo](https://mlsanity.com/paper/2305.11167/) (2023)
- [SuperNormal: Neural Surface Reconstruction via Multi-View Normal Integration](https://mlsanity.com/paper/2312.04803/) (2023)
- [ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision](https://mlsanity.com/paper/2211.14086/) (2022)
- [Multi-View Azimuth Stereo via Tangent Space Consistency](https://mlsanity.com/paper/2303.16447/) (2023)
- [HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenes](https://mlsanity.com/paper/2403.20032/) (2024)
- [Towards 3D Vision with Low-Cost Single-Photon Cameras](https://mlsanity.com/paper/2403.17801/) (2024)
- [Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos](https://mlsanity.com/paper/2403.17915/) (2024)
- [RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction](https://mlsanity.com/paper/2312.01215/) (2023)
- [Relightable and Animatable Neural Avatars from Videos](https://mlsanity.com/paper/2312.12877/) (2023)
- [Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2407.15212/) (2024)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [Relightable Neural Human Assets from Multi-view Gradient Illuminations](https://mlsanity.com/paper/2212.07648/) (2022)
- [USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields](https://mlsanity.com/paper/2310.02687/) (2023)
- [Deep Learning Methods for Calibrated Photometric Stereo and Beyond](https://mlsanity.com/paper/2212.08414/) (2022)
- [Unsupervised Multi-view Pedestrian Detection](https://mlsanity.com/paper/2305.12457/) (2023)
- [Neural Projection Mapping Using Reflectance Fields](https://mlsanity.com/paper/2306.06595/) (2023)
- [VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization](https://mlsanity.com/paper/2310.11864/) (2023)
- [Differentiable Point-based Inverse Rendering](https://mlsanity.com/paper/2312.02480/) (2023)
- [Normal-guided Detail-Preserving Neural Implicit Function for High-Fidelity 3D Surface Reconstruction](https://mlsanity.com/paper/2406.04861/) (2024)

## Abstract (arXiv)

Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then jointly optimizes the surface normals, spatially-varying BRDFs, and lights based on a shadow-aware differentiable rendering layer. After optimization, the reconstructed object can be used for novel-view rendering, relighting, and material editing. Experiments on both synthetic and real datasets demonstrate that our method achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods. Our code and model can be found at https://ywq.github.io/psnerf.