# Inferring Implicit 3D Representations from Human Figures on Pictorial Maps

arXiv: https://arxiv.org/abs/2209.02385 · HTML: https://mlsanity.com/paper/2209.02385/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 70 references that are in the map.
- First posted on arXiv 30 Aug 2022.

## Builds on (most cited first, 30 of 31)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction](https://mlsanity.com/paper/2003.10983/) (2020)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)
- [Multi-Garment Net: Learning to Dress 3D People from Images](https://mlsanity.com/paper/1908.06903/) (2019)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [AGORA: Avatars in Geography Optimized for Regression Analysis](https://mlsanity.com/paper/2104.14643/) (2021)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [BodyNet: Volumetric Inference of 3D Human Body Shapes](https://mlsanity.com/paper/1804.04875/) (2018)
- [Learning to Infer Implicit Surfaces without 3D Supervision](https://mlsanity.com/paper/1911.00767/) (2019)
- [Shape and Viewpoint without Keypoints](https://mlsanity.com/paper/2007.10982/) (2020)
- [Medical image denoising using convolutional denoising autoencoders](https://mlsanity.com/paper/1608.04667/) (2016)
- [Learning Unsupervised Hierarchical Part Decomposition of 3D Objects from a Single RGB Image](https://mlsanity.com/paper/2004.01176/) (2020)
- [SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images](https://mlsanity.com/paper/2010.10505/) (2020)
- [Shelf-Supervised Mesh Prediction in the Wild](https://mlsanity.com/paper/2102.06195/) (2021)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [3DPeople: Modeling the Geometry of Dressed Humans](https://mlsanity.com/paper/1904.04571/) (2019)
- [Im2Struct: Recovering 3D Shape Structure from a Single RGB Image](https://mlsanity.com/paper/1804.05469/) (2018)
- [D2IM-Net: Learning Detail Disentangled Implicit Fields from Single Images](https://mlsanity.com/paper/2012.06650/) (2020)
- [3D Sketching using Multi-View Deep Volumetric Prediction](https://mlsanity.com/paper/1707.08390/) (2017)
- [DenseReg: Fully Convolutional Dense Shape Regression In-the-Wild](https://mlsanity.com/paper/1803.02188/) (2018)
- [Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data](https://mlsanity.com/paper/2002.12674/) (2020)
- [DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image](https://mlsanity.com/paper/1903.10153/) (2019)
- [NeRD: Neural 3D Reflection Symmetry Detector](https://mlsanity.com/paper/2105.03211/) (2021)

## Abstract (arXiv)

In this work, we present an automated workflow to bring human figures, one of the most frequently appearing entities on pictorial maps, to the third dimension. Our workflow is based on training data and neural networks for single-view 3D reconstruction of real humans from photos. We first let a network consisting of fully connected layers estimate the depth coordinate of 2D pose points. The gained 3D pose points are inputted together with 2D masks of body parts into a deep implicit surface network to infer 3D signed distance fields (SDFs). By assembling all body parts, we derive 2D depth images and body part masks of the whole figure for different views, which are fed into a fully convolutional network to predict UV images. These UV images and the texture for the given perspective are inserted into a generative network to inpaint the textures for the other views. The textures are enhanced by a cartoonization network and facial details are resynthesized by an autoencoder. Finally, the generated textures are assigned to the inferred body parts in a ray marcher. We test our workflow with 12 pictorial human figures after having validated several network configurations. The created 3D models look generally promising, especially when considering the challenges of silhouette-based 3D recovery and real-time rendering of the implicit SDFs. Further improvement is needed to reduce gaps between the body parts and to add pictorial details to the textures. Overall, the constructed figures may be used for animation and storytelling in digital 3D maps.