# 2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning

arXiv: https://arxiv.org/abs/1802.09232 · HTML: https://mlsanity.com/paper/1802.09232/

## Key facts

- Cited by 103 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2018.

## Built on by (most cited first, 30 of 103)

- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [Putting People in their Place: Monocular Regression of 3D People in Depth](https://mlsanity.com/paper/2112.08274/) (2021)
- [Visual Imitation Enables Contextual Humanoid Control](https://mlsanity.com/paper/2505.03729/) (2025)
- [Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation](https://mlsanity.com/paper/2108.07181/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Conditional Directed Graph Convolution for 3D Human Pose Estimation](https://mlsanity.com/paper/2107.07797/) (2021)
- [Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective](https://mlsanity.com/paper/2104.11536/) (2021)
- [Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformers](https://mlsanity.com/paper/2109.07531/) (2021)
- [STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition](https://mlsanity.com/paper/2210.07503/) (2022)
- [When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks](https://mlsanity.com/paper/2105.06152/) (2021)
- [SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos](https://mlsanity.com/paper/2104.11452/) (2021)
- [Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos](https://mlsanity.com/paper/2209.09484/) (2022)
- [Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers](https://mlsanity.com/paper/2210.06110/) (2022)
- [The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions](https://mlsanity.com/paper/2104.02710/) (2021)
- [KAMA: 3D Keypoint Aware Body Mesh Articulation](https://mlsanity.com/paper/2104.13502/) (2021)
- [PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning](https://mlsanity.com/paper/2210.07886/) (2022)
- [Cross-Modal Learning with 3D Deformable Attention for Action Recognition](https://mlsanity.com/paper/2212.05638/) (2022)
- [Vision-Based Human Pose Estimation via Deep Learning: A Survey](https://mlsanity.com/paper/2308.13872/) (2023)
- [A Novel Dataset for Keypoint Detection of quadruped Animals from Images](https://mlsanity.com/paper/2108.13958/) (2021)
- [Fast Facial Landmark Detection and Applications: A Survey](https://mlsanity.com/paper/2101.10808/) (2021)
- [Image anomaly detection and prediction scheme based on SSA optimized ResNet50-BiGRU model](https://mlsanity.com/paper/2406.13987/) (2024)
- [Lifting Monocular Events to 3D Human Poses](https://mlsanity.com/paper/2104.10609/) (2021)
- [HIH: Towards More Accurate Face Alignment via Heatmap in Heatmap](https://mlsanity.com/paper/2104.03100/) (2021)
- [Subpixel Heatmap Regression for Facial Landmark Localization](https://mlsanity.com/paper/2111.02360/) (2021)
- [Reconstructing Three-Dimensional Models of Interacting Humans](https://mlsanity.com/paper/2308.01854/) (2023)
- [BaboonLand Dataset: Tracking Primates in the Wild and Automating Behaviour Recognition from Drone Videos](https://mlsanity.com/paper/2405.17698/) (2024)
- [Human Interaction Recognition Framework based on Interacting Body Part Attention](https://mlsanity.com/paper/2101.08967/) (2021)
- [Localization with Sampling-Argmax](https://mlsanity.com/paper/2110.08825/) (2021)

## Abstract (arXiv)

Action recognition and human pose estimation are closely related but both problems are generally handled as distinct tasks in the literature. In this work, we propose a multitask framework for jointly 2D and 3D pose estimation from still images and human action recognition from video sequences. We show that a single architecture can be used to solve the two problems in an efficient way and still achieves state-of-the-art results. Additionally, we demonstrate that optimization from end-to-end leads to significantly higher accuracy than separated learning. The proposed architecture can be trained with data from different categories simultaneously in a seamlessly way. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU) demonstrate the effectiveness of our method on the targeted tasks.