# Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces

arXiv: https://arxiv.org/abs/2012.06777 · HTML: https://mlsanity.com/paper/2012.06777/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2020.

## Built on by (most cited first, 30 of 33)

- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections](https://mlsanity.com/paper/2205.15768/) (2022)
- [PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo](https://mlsanity.com/paper/2207.11406/) (2022)
- [Uncertainty-Aware Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2202.13071/) (2022)
- [S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint](https://mlsanity.com/paper/2210.08936/) (2022)
- [Scalable, Detailed and Mask-Free Universal Photometric Stereo](https://mlsanity.com/paper/2303.15724/) (2023)
- [Neural Reflectance for Shape Recovery with Shadow Handling](https://mlsanity.com/paper/2203.12909/) (2022)
- [Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2110.05594/) (2021)
- [Universal Photometric Stereo Network using Global Lighting Contexts](https://mlsanity.com/paper/2206.02452/) (2022)
- [Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting](https://mlsanity.com/paper/2401.09126/) (2024)
- [SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild](https://mlsanity.com/paper/2401.10171/) (2024)
- [Self-calibrating Photometric Stereo by Neural Inverse Rendering](https://mlsanity.com/paper/2207.07815/) (2022)
- [Multi-View Photometric Stereo Revisited](https://mlsanity.com/paper/2210.07670/) (2022)
- [Fast Light-Weight Near-Field Photometric Stereo](https://mlsanity.com/paper/2203.16515/) (2022)
- [Deep Learning Methods for Calibrated Photometric Stereo and Beyond](https://mlsanity.com/paper/2212.08414/) (2022)
- [Light of Normals: Unified Feature Representation for Universal Photometric Stereo](https://mlsanity.com/paper/2506.18882/) (2025)
- [Neural Architecture Search for Efficient Uncalibrated Deep Photometric Stereo](https://mlsanity.com/paper/2110.05621/) (2021)
- [Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images](https://mlsanity.com/paper/2311.04521/) (2023)
- [SViM3D: Stable Video Material Diffusion for Single Image 3D Generation](https://mlsanity.com/paper/2510.08271/) (2025)
- [DeepShadow: Neural Shape from Shadow](https://mlsanity.com/paper/2203.15065/) (2022)
- [SupeRVol: Super-Resolution Shape and Reflectance Estimation in Inverse Volume Rendering](https://mlsanity.com/paper/2212.04968/) (2022)
- [DeepPS2: Revisiting Photometric Stereo Using Two Differently Illuminated Images](https://mlsanity.com/paper/2207.02025/) (2022)
- [Multi-view Surface Reconstruction Using Normal and Reflectance Cues](https://mlsanity.com/paper/2506.04115/) (2025)
- [MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo](https://mlsanity.com/paper/2409.00674/) (2024)
- [Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition](https://mlsanity.com/paper/2410.20716/) (2024)
- [MS-PS: A Multi-Scale Network for Photometric Stereo With a New Comprehensive Training Dataset](https://mlsanity.com/paper/2211.14118/) (2022)
- [Mobile Robotic Multi-View Photometric Stereo](https://mlsanity.com/paper/2502.10842/) (2025)
- [Endoscopic Depth Estimation Based on Deep Learning: A Survey](https://mlsanity.com/paper/2507.20881/) (2025)
- [Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues](https://mlsanity.com/paper/2511.13015/) (2025)
- [Deep Uncalibrated Photometric Stereo via Inter-Intra Image Feature Fusion](https://mlsanity.com/paper/2208.03440/) (2022)

## Abstract (arXiv)

This paper presents an uncalibrated deep neural network framework for the photometric stereo problem. For training models to solve the problem, existing neural network-based methods either require exact light directions or ground-truth surface normals of the object or both. However, in practice, it is challenging to procure both of this information precisely, which restricts the broader adoption of photometric stereo algorithms for vision application. To bypass this difficulty, we propose an uncalibrated neural inverse rendering approach to this problem. Our method first estimates the light directions from the input images and then optimizes an image reconstruction loss to calculate the surface normals, bidirectional reflectance distribution function value, and depth. Additionally, our formulation explicitly models the concave and convex parts of a complex surface to consider the effects of interreflections in the image formation process. Extensive evaluation of the proposed method on the challenging subjects generally shows comparable or better results than the supervised and classical approaches.