# Neural Inverse Rendering for General Reflectance Photometric Stereo

arXiv: https://arxiv.org/abs/1802.10328 · HTML: https://mlsanity.com/paper/1802.10328/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2018.

## Built on by (most cited first, 30 of 35)

- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [Uncertainty-Aware Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2202.13071/) (2022)
- [S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint](https://mlsanity.com/paper/2210.08936/) (2022)
- [Scalable, Detailed and Mask-Free Universal Photometric Stereo](https://mlsanity.com/paper/2303.15724/) (2023)
- [Neural Reflectance for Shape Recovery with Shadow Handling](https://mlsanity.com/paper/2203.12909/) (2022)
- [Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2110.05594/) (2021)
- [GenLit: Reformulating Single-Image Relighting as Video Generation](https://mlsanity.com/paper/2412.11224/) (2024)
- [Universal Photometric Stereo Network using Global Lighting Contexts](https://mlsanity.com/paper/2206.02452/) (2022)
- [Self-calibrating Photometric Stereo by Neural Inverse Rendering](https://mlsanity.com/paper/2207.07815/) (2022)
- [A CNN Based Approach for the Point-Light Photometric Stereo Problem](https://mlsanity.com/paper/2210.04655/) (2022)
- [Deep Learning Methods for Calibrated Photometric Stereo and Beyond](https://mlsanity.com/paper/2212.08414/) (2022)
- [Light of Normals: Unified Feature Representation for Universal Photometric Stereo](https://mlsanity.com/paper/2506.18882/) (2025)
- [Neural Architecture Search for Efficient Uncalibrated Deep Photometric Stereo](https://mlsanity.com/paper/2110.05621/) (2021)
- [DeepShadow: Neural Shape from Shadow](https://mlsanity.com/paper/2203.15065/) (2022)
- [Edge-preserving Near-light Photometric Stereo with Neural Surfaces](https://mlsanity.com/paper/2207.04622/) (2022)
- [SupeRVol: Super-Resolution Shape and Reflectance Estimation in Inverse Volume Rendering](https://mlsanity.com/paper/2212.04968/) (2022)
- [DeepPS2: Revisiting Photometric Stereo Using Two Differently Illuminated Images](https://mlsanity.com/paper/2207.02025/) (2022)
- [DANI-Net: Uncalibrated Photometric Stereo by Differentiable Shadow Handling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering](https://mlsanity.com/paper/2303.15101/) (2023)
- [Multi-view Surface Reconstruction Using Normal and Reflectance Cues](https://mlsanity.com/paper/2506.04115/) (2025)
- [Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo](https://mlsanity.com/paper/2103.12106/) (2021)
- [Incorporating Lambertian Priors into Surface Normals Measurement](https://mlsanity.com/paper/2107.07192/) (2021)
- [Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition](https://mlsanity.com/paper/2410.20716/) (2024)
- [MS-PS: A Multi-Scale Network for Photometric Stereo With a New Comprehensive Training Dataset](https://mlsanity.com/paper/2211.14118/) (2022)
- [An Optimal Experimental Design Approach for Light Configurations in Photometric Stereo](https://mlsanity.com/paper/2204.05218/) (2022)
- [RMAFF-PSN: A Residual Multi-Scale Attention Feature Fusion Photometric Stereo Network](https://mlsanity.com/paper/2404.07766/) (2024)
- [Image Gradient-Aided Photometric Stereo Network](https://mlsanity.com/paper/2412.11650/) (2024)
- [Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues](https://mlsanity.com/paper/2511.13015/) (2025)
- [Imperceptible Adversarial Examples for Fake Image Detection](https://mlsanity.com/paper/2106.01615/) (2021)
- [NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo](https://mlsanity.com/paper/2208.08897/) (2022)

## Abstract (arXiv)

We present a novel convolutional neural network architecture for photometric stereo (Woodham, 1980), a problem of recovering 3D object surface normals from multiple images observed under varying illuminations. Despite its long history in computer vision, the problem still shows fundamental challenges for surfaces with unknown general reflectance properties (BRDFs). Leveraging deep neural networks to learn complicated reflectance models is promising, but studies in this direction are very limited due to difficulties in acquiring accurate ground truth for training and also in designing networks invariant to permutation of input images. In order to address these challenges, we propose a physics based unsupervised learning framework where surface normals and BRDFs are predicted by the network and fed into the rendering equation to synthesize observed images. The network weights are optimized during testing by minimizing reconstruction loss between observed and synthesized images. Thus, our learning process does not require ground truth normals or even pre-training on external images. Our method is shown to achieve the state-of-the-art performance on a challenging real-world scene benchmark.