# StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision

arXiv: https://arxiv.org/abs/2104.05289 · HTML: https://mlsanity.com/paper/2104.05289/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction](https://mlsanity.com/paper/2007.03858/) (2020)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [AANet: Adaptive Aggregation Network for Efficient Stereo Matching](https://mlsanity.com/paper/2004.09548/) (2020)
- [DeepHuman: 3D Human Reconstruction from a Single Image](https://mlsanity.com/paper/1903.06473/) (2019)
- [Learning to Reconstruct People in Clothing from a Single RGB Camera](https://mlsanity.com/paper/1903.05885/) (2019)
- [Tex2Shape: Detailed Full Human Body Geometry From a Single Image](https://mlsanity.com/paper/1904.08645/) (2019)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)

## Built on by (most cited first, 30 of 38)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis](https://mlsanity.com/paper/2312.02155/) (2023)
- [SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video](https://mlsanity.com/paper/2201.12792/) (2022)
- [High-fidelity 3D Human Digitization from Single 2K Resolution Images](https://mlsanity.com/paper/2303.15108/) (2023)
- [DoubleField: Bridging the Neural Surface and Radiance Fields for High-fidelity Human Reconstruction and Rendering](https://mlsanity.com/paper/2106.03798/) (2021)
- [DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras](https://mlsanity.com/paper/2207.08000/) (2022)
- [MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos](https://mlsanity.com/paper/2407.08414/) (2024)
- [REC-MV: REconstructing 3D Dynamic Cloth from Monocular Videos](https://mlsanity.com/paper/2305.14236/) (2023)
- [D-IF: Uncertainty-aware Human Digitization via Implicit Distribution Field](https://mlsanity.com/paper/2308.08857/) (2023)
- [AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture](https://mlsanity.com/paper/2207.02031/) (2022)
- [GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views](https://mlsanity.com/paper/2411.11363/) (2024)
- [MonoNHR: Monocular Neural Human Renderer](https://mlsanity.com/paper/2210.00627/) (2022)
- [SeSDF: Self-evolved Signed Distance Field for Implicit 3D Clothed Human Reconstruction](https://mlsanity.com/paper/2304.00359/) (2023)
- [Rendering Humans from Object-Occluded Monocular Videos](https://mlsanity.com/paper/2308.04622/) (2023)
- [A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generation](https://mlsanity.com/paper/2406.04253/) (2024)
- [Learning Visibility Field for Detailed 3D Human Reconstruction and Relighting](https://mlsanity.com/paper/2304.11900/) (2023)
- [SelfNeRF: Fast Training NeRF for Human from Monocular Self-rotating Video](https://mlsanity.com/paper/2210.01651/) (2022)
- [IntegratedPIFu: Integrated Pixel Aligned Implicit Function for Single-view Human Reconstruction](https://mlsanity.com/paper/2211.07955/) (2022)
- [Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image](https://mlsanity.com/paper/2211.11931/) (2022)
- [Neural-ABC: Neural Parametric Models for Articulated Body with Clothes](https://mlsanity.com/paper/2404.04673/) (2024)
- [IntrinsicNGP: Intrinsic Coordinate based Hash Encoding for Human NeRF](https://mlsanity.com/paper/2302.14683/) (2023)
- [Super-resolution 3D Human Shape from a Single Low-Resolution Image](https://mlsanity.com/paper/2208.10738/) (2022)
- [High-Resolution Volumetric Reconstruction for Clothed Humans](https://mlsanity.com/paper/2307.13282/) (2023)
- [High-Quality Animatable Dynamic Garment Reconstruction from Monocular Videos](https://mlsanity.com/paper/2311.01214/) (2023)
- [Geometry-aware Two-scale PIFu Representation for Human Reconstruction](https://mlsanity.com/paper/2112.02082/) (2021)
- [Occupancy Planes for Single-view RGB-D Human Reconstruction](https://mlsanity.com/paper/2208.02817/) (2022)
- [A Generative Multi-Resolution Pyramid and Normal-Conditioning 3D Cloth Draping](https://mlsanity.com/paper/2311.02700/) (2023)
- [USR: Unsupervised Separated 3D Garment and Human Reconstruction via Geometry and Semantic Consistency](https://mlsanity.com/paper/2302.10518/) (2023)
- [MVP-Human Dataset for 3D Human Avatar Reconstruction from Unconstrained Frames](https://mlsanity.com/paper/2204.11184/) (2022)

## Abstract (arXiv)

In this paper, we propose StereoPIFu, which integrates the geometric constraints of stereo vision with implicit function representation of PIFu, to recover the 3D shape of the clothed human from a pair of low-cost rectified images. First, we introduce the effective voxel-aligned features from a stereo vision-based network to enable depth-aware reconstruction. Moreover, the novel relative z-offset is employed to associate predicted high-fidelity human depth and occupancy inference, which helps restore fine-level surface details. Second, a network structure that fully utilizes the geometry information from the stereo images is designed to improve the human body reconstruction quality. Consequently, our StereoPIFu can naturally infer the human body's spatial location in camera space and maintain the correct relative position of different parts of the human body, which enables our method to capture human performance. Compared with previous works, our StereoPIFu significantly improves the robustness, completeness, and accuracy of the clothed human reconstruction, which is demonstrated by extensive experimental results.