# PS-Transformer: Learning Sparse Photometric Stereo Network using Self-Attention Mechanism

arXiv: https://arxiv.org/abs/2211.11386 · HTML: https://mlsanity.com/paper/2211.11386/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](https://mlsanity.com/paper/1810.00825/) (2018)
- [Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials](https://mlsanity.com/paper/2001.06659/) (2020)
- [PS-FCN: A Flexible Learning Framework for Photometric Stereo](https://mlsanity.com/paper/1807.08696/) (2018)
- [Self-calibrating Deep Photometric Stereo Networks](https://mlsanity.com/paper/1903.07366/) (2019)
- [CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces](https://mlsanity.com/paper/1808.10093/) (2018)
- [SPLINE-Net: Sparse Photometric Stereo through Lighting Interpolation and Normal Estimation Networks](https://mlsanity.com/paper/1905.04088/) (2019)
- [Deep Photometric Stereo for Non-Lambertian Surfaces](https://mlsanity.com/paper/2007.13145/) (2020)
- [Shape and Spatially-Varying Reflectance Estimation From Virtual Exemplars](https://mlsanity.com/paper/1512.05278/) (2015)

## Built on by (most cited first, 13 of 13)

- [Scalable, Detailed and Mask-Free Universal Photometric Stereo](https://mlsanity.com/paper/2303.15724/) (2023)
- [MVPSNet: Fast Generalizable Multi-view Photometric Stereo](https://mlsanity.com/paper/2305.11167/) (2023)
- [Deep Learning Methods for Calibrated Photometric Stereo and Beyond](https://mlsanity.com/paper/2212.08414/) (2022)
- [Light of Normals: Unified Feature Representation for Universal Photometric Stereo](https://mlsanity.com/paper/2506.18882/) (2025)
- [Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition](https://mlsanity.com/paper/2410.20716/) (2024)
- [A Real-world Display Inverse Rendering Dataset](https://mlsanity.com/paper/2508.14411/) (2025)
- [MS-PS: A Multi-Scale Network for Photometric Stereo With a New Comprehensive Training Dataset](https://mlsanity.com/paper/2211.14118/) (2022)
- [Image Gradient-Aided Photometric Stereo Network](https://mlsanity.com/paper/2412.11650/) (2024)
- [Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues](https://mlsanity.com/paper/2511.13015/) (2025)
- [Open-World Object Detection via Discriminative Class Prototype Learning](https://mlsanity.com/paper/2302.11757/) (2023)
- [LIPIDS: Learning-based Illumination Planning In Discretized (Light) Space for Photometric Stereo](https://mlsanity.com/paper/2409.02716/) (2024)
- [PS-EIP: Robust Photometric Stereo Based on Event Interval Profile](https://mlsanity.com/paper/2503.18341/) (2025)
- [MSF-Net: Multi-Stage Feature Extraction and Fusion for Robust Photometric Stereo](https://mlsanity.com/paper/2510.25221/) (2025)

## Abstract (arXiv)

Existing deep calibrated photometric stereo networks basically aggregate observations under different lights based on the pre-defined operations such as linear projection and max pooling. While they are effective with the dense capture, simple first-order operations often fail to capture the high-order interactions among observations under small number of different lights. To tackle this issue, this paper presents a deep sparse calibrated photometric stereo network named it PS-Transformer which leverages the learnable self-attention mechanism to properly capture the complex inter-image interactions. PS-Transformer builds upon the dual-branch design to explore both pixel-wise and image-wise features and individual feature is trained with the intermediate surface normal supervision to maximize geometric feasibility. A new synthetic dataset named CyclesPS+ is also presented with the comprehensive analysis to successfully train the photometric stereo networks. Extensive results on the publicly available benchmark datasets demonstrate that the surface normal prediction accuracy of the proposed method significantly outperforms other state-of-the-art algorithms with the same number of input images and is even comparable to that of dense algorithms which input 10× larger number of images.