# Deep Uncalibrated Photometric Stereo via Inter-Intra Image Feature Fusion

arXiv: https://arxiv.org/abs/2208.03440 · HTML: https://mlsanity.com/paper/2208.03440/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 35 references that are in the map.
- First posted on arXiv 6 Aug 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [PS-FCN: A Flexible Learning Framework for Photometric Stereo](https://mlsanity.com/paper/1807.08696/) (2018)
- [Flexible SVBRDF Capture with a Multi-Image Deep Network](https://mlsanity.com/paper/1906.11557/) (2019)
- [Self-calibrating Deep Photometric Stereo Networks](https://mlsanity.com/paper/1903.07366/) (2019)
- [Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces](https://mlsanity.com/paper/2012.06777/) (2020)
- [Monocular Dense 3D Reconstruction of a Complex Dynamic Scene from Two Perspective Frames](https://mlsanity.com/paper/1708.04398/) (2017)
- [SilNet : Single- and Multi-View Reconstruction by Learning from Silhouettes](https://mlsanity.com/paper/1711.07888/) (2017)
- [Superpixel Soup: Monocular Dense 3D Reconstruction of a Complex Dynamic Scene](https://mlsanity.com/paper/1911.09092/) (2019)

## Abstract (arXiv)

Uncalibrated photometric stereo is proposed to estimate the detailed surface normal from images under varying and unknown lightings. Recently, deep learning brings powerful data priors to this underdetermined problem. This paper presents a new method for deep uncalibrated photometric stereo, which efficiently utilizes the inter-image representation to guide the normal estimation. Previous methods use optimization-based neural inverse rendering or a single size-independent pooling layer to deal with multiple inputs, which are inefficient for utilizing information among input images. Given multi-images under different lighting, we consider the intra-image and inter-image variations highly correlated. Motivated by the correlated variations, we designed an inter-intra image feature fusion module to introduce the inter-image representation into the per-image feature extraction. The extra representation is used to guide the per-image feature extraction and eliminate the ambiguity in normal estimation. We demonstrate the effect of our design on a wide range of samples, especially on dark materials. Our method produces significantly better results than the state-of-the-art methods on both synthetic and real data.