# Scalable, Detailed and Mask-Free Universal Photometric Stereo

arXiv: https://arxiv.org/abs/2303.15724 · HTML: https://mlsanity.com/paper/2303.15724/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 71 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Modeling Indirect Illumination for Inverse Rendering](https://mlsanity.com/paper/2204.06837/) (2022)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022)
- [PS-FCN: A Flexible Learning Framework for Photometric Stereo](https://mlsanity.com/paper/1807.08696/) (2018)
- [Self-calibrating Deep Photometric Stereo Networks](https://mlsanity.com/paper/1903.07366/) (2019)
- [CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces](https://mlsanity.com/paper/1808.10093/) (2018)
- [Neural Inverse Rendering for General Reflectance Photometric Stereo](https://mlsanity.com/paper/1802.10328/) (2018)
- [Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces](https://mlsanity.com/paper/2012.06777/) (2020)
- [Neural Reflectance for Shape Recovery with Shadow Handling](https://mlsanity.com/paper/2203.12909/) (2022)
- [PX-NET: Simple and Efficient Pixel-Wise Training of Photometric Stereo Networks](https://mlsanity.com/paper/2008.04933/) (2020)
- [Universal Photometric Stereo Network using Global Lighting Contexts](https://mlsanity.com/paper/2206.02452/) (2022)
- [Self-calibrating Photometric Stereo by Neural Inverse Rendering](https://mlsanity.com/paper/2207.07815/) (2022)
- [Shape and Spatially-Varying Reflectance Estimation From Virtual Exemplars](https://mlsanity.com/paper/1512.05278/) (2015)
- [Variational Uncalibrated Photometric Stereo under General Lighting](https://mlsanity.com/paper/1904.03942/) (2019)
- [PS-Transformer: Learning Sparse Photometric Stereo Network using Self-Attention Mechanism](https://mlsanity.com/paper/2211.11386/) (2022)
- [DeepPS2: Revisiting Photometric Stereo Using Two Differently Illuminated Images](https://mlsanity.com/paper/2207.02025/) (2022)
- [Single Day Outdoor Photometric Stereo](https://mlsanity.com/paper/1803.10850/) (2018)

## Built on by (most cited first, 30 of 30)

- [Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridging](https://mlsanity.com/paper/2503.22236/) (2025)
- [IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations](https://mlsanity.com/paper/2412.12083/) (2024)
- [SuperNormal: Neural Surface Reconstruction via Multi-View Normal Integration](https://mlsanity.com/paper/2312.04803/) (2023)
- [RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction](https://mlsanity.com/paper/2312.01215/) (2023)
- [Light of Normals: Unified Feature Representation for Universal Photometric Stereo](https://mlsanity.com/paper/2506.18882/) (2025)
- [Multi-view Surface Reconstruction Using Normal and Reflectance Cues](https://mlsanity.com/paper/2506.04115/) (2025)
- [Differentiable Display Photometric Stereo](https://mlsanity.com/paper/2306.13325/) (2023)
- [NPLMV-PS: Neural Point-Light Multi-View Photometric Stereo](https://mlsanity.com/paper/2405.12057/) (2024)
- [MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo](https://mlsanity.com/paper/2409.00674/) (2024)
- [Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition](https://mlsanity.com/paper/2410.20716/) (2024)
- [PRM: Photometric Stereo based Large Reconstruction Model](https://mlsanity.com/paper/2412.07371/) (2024)
- [A Real-world Display Inverse Rendering Dataset](https://mlsanity.com/paper/2508.14411/) (2025)
- [LUCES-MV: A Multi-View Dataset for Near-Field Point Light Source Photometric Stereo](https://mlsanity.com/paper/2412.16737/) (2024)
- [Monocular Normal Estimation via Shading Sequence Estimation](https://mlsanity.com/paper/2602.09929/) (2026)
- [Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo](https://mlsanity.com/paper/2404.01612/) (2024)
- [PMNI: Pose-free Multi-view Normal Integration for Reflective and Textureless Surface Reconstruction](https://mlsanity.com/paper/2504.08410/) (2025)
- [Discontinuity-aware Normal Integration for Generic Central Camera Models](https://mlsanity.com/paper/2507.06075/) (2025)
- [PS-GS: Gaussian Splatting for Multi-View Photometric Stereo](https://mlsanity.com/paper/2507.18231/) (2025)
- [Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues](https://mlsanity.com/paper/2511.13015/) (2025)
- [LIPIDS: Learning-based Illumination Planning In Discretized (Light) Space for Photometric Stereo](https://mlsanity.com/paper/2409.02716/) (2024)
- [PS-EIP: Robust Photometric Stereo Based on Event Interval Profile](https://mlsanity.com/paper/2503.18341/) (2025)
- [Neural Multi-View Self-Calibrated Photometric Stereo without Photometric Stereo Cues](https://mlsanity.com/paper/2507.23162/) (2025)
- [Towards Fast and Scalable Normal Integration using Continuous Components](https://mlsanity.com/paper/2510.11508/) (2025)
- [Near-Light Color Photometric Stereo for Mono-Chromatic Non-Lambertian Surfaces](https://mlsanity.com/paper/2601.12666/) (2026)
- [EvalMVX: A Unified Benchmarking for Neural 3D Reconstruction under Diverse Multiview Setups](https://mlsanity.com/paper/2602.24065/) (2026)
- [High-fidelity Multi-view Normal Integration with Scale-encoded Neural Surface Representation](https://mlsanity.com/paper/2603.20337/) (2026)
- [Snapshot Polarimetric Display Inverse Rendering](https://mlsanity.com/paper/2605.24915/) (2026)
- [FlashNormal: Detailed Surface Normal Estimation from Flash and No-Flash Images](https://mlsanity.com/paper/2608.25360/) (2026)
- [Integrating Multi-view Multi-light Surface Reconstruction into Cultural Heritage Workflows](https://mlsanity.com/paper/2609.15833/) (2026)
- [Calibration-Free Surface Normals Estimation in Vision-Based Tactile Sensing using Universal Photometric Stereo](https://mlsanity.com/paper/2609.31754/) (2026)

## Abstract (arXiv)

In this paper, we introduce SDM-UniPS, a groundbreaking Scalable, Detailed, Mask-free, and Universal Photometric Stereo network. Our approach can recover astonishingly intricate surface normal maps, rivaling the quality of 3D scanners, even when images are captured under unknown, spatially-varying lighting conditions in uncontrolled environments. We have extended previous universal photometric stereo networks to extract spatial-light features, utilizing all available information in high-resolution input images and accounting for non-local interactions among surface points. Moreover, we present a new synthetic training dataset that encompasses a diverse range of shapes, materials, and illumination scenarios found in real-world scenes. Through extensive evaluation, we demonstrate that our method not only surpasses calibrated, lighting-specific techniques on public benchmarks, but also excels with a significantly smaller number of input images even without object masks.