# Deep Multitask Architecture for Integrated 2D and 3D Human Sensing

arXiv: https://arxiv.org/abs/1701.08985 · HTML: https://mlsanity.com/paper/1701.08985/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2017.

## Built on by (most cited first, 21 of 21)

- [Direct Multi-view Multi-person 3D Pose Estimation](https://mlsanity.com/paper/2111.04076/) (2021)
- [THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers](https://mlsanity.com/paper/2106.09336/) (2021)
- [Body Meshes as Points](https://mlsanity.com/paper/2105.02467/) (2021)
- [Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration](https://mlsanity.com/paper/2104.08160/) (2021)
- [Hand Image Understanding via Deep Multi-Task Learning](https://mlsanity.com/paper/2107.11646/) (2021)
- [HSPACE: Synthetic Parametric Humans Animated in Complex Environments](https://mlsanity.com/paper/2112.12867/) (2021)
- [KAMA: 3D Keypoint Aware Body Mesh Articulation](https://mlsanity.com/paper/2104.13502/) (2021)
- [SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation](https://mlsanity.com/paper/2404.02041/) (2024)
- [3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers](https://mlsanity.com/paper/2110.08861/) (2021)
- [Reconstructing Three-Dimensional Models of Interacting Humans](https://mlsanity.com/paper/2308.01854/) (2023)
- [HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances](https://mlsanity.com/paper/2210.05665/) (2022)
- [On Triangulation as a Form of Self-Supervision for 3D Human Pose Estimation](https://mlsanity.com/paper/2203.15865/) (2022)
- [VirtualPose: Learning Generalizable 3D Human Pose Models from Virtual Data](https://mlsanity.com/paper/2207.09949/) (2022)
- [Distribution-Aware Single-Stage Models for Multi-Person 3D Pose Estimation](https://mlsanity.com/paper/2203.07697/) (2022)
- [Dynamic Graph Reasoning for Multi-person 3D Pose Estimation](https://mlsanity.com/paper/2207.11341/) (2022)
- [Enhancing Egocentric 3D Pose Estimation with Third Person Views](https://mlsanity.com/paper/2201.02017/) (2022)
- [Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation](https://mlsanity.com/paper/2309.11667/) (2023)
- [Adapted Human Pose: Monocular 3D Human Pose Estimation with Zero Real 3D Pose Data](https://mlsanity.com/paper/2105.10837/) (2021)
- [Deep3DPose: Realtime Reconstruction of Arbitrarily Posed Human Bodies from Single RGB Images](https://mlsanity.com/paper/2106.11536/) (2021)
- [Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle](https://mlsanity.com/paper/2309.06313/) (2023)
- [Occlusion Resilient 3D Human Pose Estimation](https://mlsanity.com/paper/2402.11036/) (2024)

## Abstract (arXiv)

We propose a deep multitask architecture for fully automatic 2d and 3d human sensing (DMHS), including recognition and reconstruction, in monocular images. The system computes the figure-ground segmentation, semantically identifies the human body parts at pixel level, and estimates the 2d and 3d pose of the person. The model supports the joint training of all components by means of multi-task losses where early processing stages recursively feed into advanced ones for increasingly complex calculations, accuracy and robustness. The design allows us to tie a complete training protocol, by taking advantage of multiple datasets that would otherwise restrictively cover only some of the model components: complex 2d image data with no body part labeling and without associated 3d ground truth, or complex 3d data with limited 2d background variability. In detailed experiments based on several challenging 2d and 3d datasets (LSP, HumanEva, Human3.6M), we evaluate the sub-structures of the model, the effect of various types of training data in the multitask loss, and demonstrate that state-of-the-art results can be achieved at all processing levels. We also show that in the wild our monocular RGB architecture is perceptually competitive to a state-of-the art (commercial) Kinect system based on RGB-D data.