# Differentiable Display Photometric Stereo

arXiv: https://arxiv.org/abs/2306.13325 · HTML: https://mlsanity.com/paper/2306.13325/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation](https://mlsanity.com/paper/2109.09881/) (2021)
- [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022)
- [Neural Light Transport for Relighting and View Synthesis](https://mlsanity.com/paper/2008.03806/) (2020)
- [Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials](https://mlsanity.com/paper/2001.06659/) (2020)
- [Deep Polarization Imaging for 3D shape and SVBRDF Acquisition](https://mlsanity.com/paper/2105.02875/) (2021)
- [Scalable, Detailed and Mask-Free Universal Photometric Stereo](https://mlsanity.com/paper/2303.15724/) (2023)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [A Light Stage on Every Desk](https://mlsanity.com/paper/2105.08051/) (2021)
- [High-Res Facial Appearance Capture from Polarized Smartphone Images](https://mlsanity.com/paper/2212.01160/) (2022)
- [WildLight: In-the-wild Inverse Rendering with a Flashlight](https://mlsanity.com/paper/2303.14190/) (2023)
- [Universal Photometric Stereo Network using Global Lighting Contexts](https://mlsanity.com/paper/2206.02452/) (2022)
- [LUCES: A Dataset for Near-Field Point Light Source Photometric Stereo](https://mlsanity.com/paper/2104.13135/) (2021)
- [From Shading to Local Shape](https://mlsanity.com/paper/1310.2916/) (2013)
- [All-photon Polarimetric Time-of-Flight Imaging](https://mlsanity.com/paper/2112.09278/) (2021)
- [Relightable Neural Human Assets from Multi-view Gradient Illuminations](https://mlsanity.com/paper/2212.07648/) (2022)

## Built on by (most cited first, 4 of 4)

- [A Real-world Display Inverse Rendering Dataset](https://mlsanity.com/paper/2508.14411/) (2025)
- [Monocular Facial Appearance Capture in the Wild](https://mlsanity.com/paper/2412.12765/) (2024)
- [Differentiable Adaptive 4D Structured Illumination for Joint Capture of Shape and Reflectance](https://mlsanity.com/paper/2605.06214/) (2026)
- [Snapshot Polarimetric Display Inverse Rendering](https://mlsanity.com/paper/2605.24915/) (2026)

## Abstract (arXiv)

Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. Display photometric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we present differentiable display photometric stereo (DDPS), addressing an often overlooked challenge in display photometric stereo: the design of display patterns. Departing from using heuristic display patterns, DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner. To this end, we propose a differentiable framework that couples basis-illumination image formation with analytic photometric-stereo reconstruction. The differentiable framework facilitates the effective learning of display patterns via auto-differentiation. Also, for training supervision, we propose to use 3D printing for creating a real-world training dataset, enabling accurate reconstruction on the target real-world setup. Finally, we exploit that conventional LCD monitors emit polarized light, which allows for the optical separation of diffuse and specular reflections when combined with a polarization camera, leading to accurate normal reconstruction. Extensive evaluation of DDPS shows improved normal-reconstruction accuracy compared to heuristic patterns and demonstrates compelling properties such as robustness to pattern initialization, calibration errors, and simplifications in image formation and reconstruction.