# Shape from Polarization for Complex Scenes in the Wild

arXiv: https://arxiv.org/abs/2112.11377 · HTML: https://mlsanity.com/paper/2112.11377/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018)
- [Polarized Reflection Removal with Perfect Alignment in the Wild](https://mlsanity.com/paper/2003.12789/) (2020)
- [SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection](https://mlsanity.com/paper/2008.11351/) (2020)
- [Designing Deep Networks for Surface Normal Estimation](https://mlsanity.com/paper/1411.4958/) (2014)
- [Deep Polarization Imaging for 3D shape and SVBRDF Acquisition](https://mlsanity.com/paper/2105.02875/) (2021)
- [Self-calibrating Deep Photometric Stereo Networks](https://mlsanity.com/paper/1903.07366/) (2019)
- [GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation](https://mlsanity.com/paper/2012.06980/) (2020)
- [Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks](https://mlsanity.com/paper/1612.07429/) (2016)
- [Surface Normal Estimation of Tilted Images via Spatial Rectifier](https://mlsanity.com/paper/2007.09264/) (2020)
- [Depth from a polarisation + RGB stereo pair](https://mlsanity.com/paper/1903.12061/) (2019)
- [Polarimetric Multi-View Inverse Rendering](https://mlsanity.com/paper/2007.08830/) (2020)
- [FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB Image](https://mlsanity.com/paper/1903.12305/) (2019)
- [Deep Photometric Stereo for Non-Lambertian Surfaces](https://mlsanity.com/paper/2007.13145/) (2020)
- [PixelNet: Towards a General Pixel-level Architecture](https://mlsanity.com/paper/1609.06694/) (2016)

## Built on by (most cited first, 30 of 53)

- [HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios](https://mlsanity.com/paper/2212.10428/) (2022)
- [PANDORA: Polarization-Aided Neural Decomposition Of Radiance](https://mlsanity.com/paper/2203.13458/) (2022)
- [NeISF: Neural Incident Stokes Field for Geometry and Material Estimation](https://mlsanity.com/paper/2311.13187/) (2023)
- [NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images](https://mlsanity.com/paper/2406.07111/) (2024)
- [Transparent Shape from a Single View Polarization Image](https://mlsanity.com/paper/2204.06331/) (2022)
- [Event-based Shape from Polarization](https://mlsanity.com/paper/2301.06855/) (2023)
- [PISR: Polarimetric Neural Implicit Surface Reconstruction for Textureless and Specular Objects](https://mlsanity.com/paper/2409.14331/) (2024)
- [Perspective Phase Angle Model for Polarimetric 3D Reconstruction](https://mlsanity.com/paper/2207.09629/) (2022)
- [Neural Spectro-polarimetric Fields](https://mlsanity.com/paper/2306.12562/) (2023)
- [Simultaneous Acquisition of High Quality RGB Image and Polarization Information using a Sparse Polarization Sensor](https://mlsanity.com/paper/2209.13106/) (2022)
- [Spectral and Polarization Vision: Spectro-polarimetric Real-world Dataset](https://mlsanity.com/paper/2311.17396/) (2023)
- [PolGS: Polarimetric Gaussian Splatting for Fast Reflective Surface Reconstruction](https://mlsanity.com/paper/2509.19726/) (2025)
- [Sparse Ellipsometry: Portable Acquisition of Polarimetric SVBRDF and Shape with Unstructured Flash Photography](https://mlsanity.com/paper/2207.04236/) (2022)
- [ShareCMP: Polarization-Aware RGB-P Semantic Segmentation](https://mlsanity.com/paper/2312.03430/) (2023)
- [Polarization Wavefront Lidar: Learning Large Scene Reconstruction from Polarized Wavefronts](https://mlsanity.com/paper/2406.03461/) (2024)
- [Robust Depth Enhancement via Polarization Prompt Fusion Tuning](https://mlsanity.com/paper/2404.04318/) (2024)
- [Deep Polarization Cues for Single-shot Shape and Subsurface Scattering Estimation](https://mlsanity.com/paper/2407.08149/) (2024)
- [PolarAnything: Diffusion-based Polarimetric Image Synthesis](https://mlsanity.com/paper/2507.17268/) (2025)
- [Polarimetric Imaging for Perception](https://mlsanity.com/paper/2305.14787/) (2023)
- [SS-SfP:Neural Inverse Rendering for Self Supervised Shape from (Mixed) Polarization](https://mlsanity.com/paper/2407.09294/) (2024)
- [Polarimetric BSSRDF Acquisition of Dynamic Faces](https://mlsanity.com/paper/2501.01980/) (2024)
- [Benchmarking Burst Super-Resolution for Polarization Images: Noise Dataset and Analysis](https://mlsanity.com/paper/2503.18705/) (2025)
- [Fooling Polarization-based Vision using Locally Controllable Polarizing Projection](https://mlsanity.com/paper/2303.17890/) (2023)
- [S2P3: Self-Supervised Polarimetric Pose Prediction](https://mlsanity.com/paper/2312.01105/) (2023)
- [NeST: Neural Stress Tensor Tomography by leveraging 3D Photoelasticity](https://mlsanity.com/paper/2406.10212/) (2024)
- [Shape from Polarization of Thermal Emission and Reflection](https://mlsanity.com/paper/2506.18217/) (2025)
- [Thermal Polarimetric Multi-view Stereo](https://mlsanity.com/paper/2510.20972/) (2025)
- [Stokes-Informed Diffusion for Robust Linear Polarization Estimation](https://mlsanity.com/paper/2607.21239/) (2026)
- [Pola4All: survey of polarimetric applications and an open-source toolkit to analyze polarization](https://mlsanity.com/paper/2312.14697/) (2023)
- [Event-based Shape from Polarization with Spiking Neural Networks](https://mlsanity.com/paper/2312.16071/) (2023)

## Abstract (arXiv)

We present a new data-driven approach with physics-based priors to scene-level normal estimation from a single polarization image. Existing shape from polarization (SfP) works mainly focus on estimating the normal of a single object rather than complex scenes in the wild. A key barrier to high-quality scene-level SfP is the lack of real-world SfP data in complex scenes. Hence, we contribute the first real-world scene-level SfP dataset with paired input polarization images and ground-truth normal maps. Then we propose a learning-based framework with a multi-head self-attention module and viewing encoding, which is designed to handle increasing polarization ambiguities caused by complex materials and non-orthographic projection in scene-level SfP. Our trained model can be generalized to far-field outdoor scenes as the relationship between polarized light and surface normals is not affected by distance. Experimental results demonstrate that our approach significantly outperforms existing SfP models on two datasets. Our dataset and source code will be publicly available at https://github.com/ChenyangLEI/sfp-wild