# Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

arXiv: https://arxiv.org/abs/1912.08077 · HTML: https://mlsanity.com/paper/1912.08077/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2019.

## Built on by (most cited first, 30 of 35)

- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition](https://mlsanity.com/paper/2107.08580/) (2021)
- [Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition](https://mlsanity.com/paper/2107.00606/) (2021)
- [Adaptive Multi-view and Temporal Fusing Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2110.05092/) (2021)
- [Egocentric Video Task Translation](https://mlsanity.com/paper/2212.06301/) (2022)
- [EAGLE: Egocentric AGgregated Language-video Engine](https://mlsanity.com/paper/2409.17523/) (2024)
- [Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation](https://mlsanity.com/paper/2311.12028/) (2023)
- [Dual networks based 3D Multi-Person Pose Estimation from Monocular Video](https://mlsanity.com/paper/2205.00748/) (2022)
- [Deep learning for 3D human pose estimation and mesh recovery: A survey](https://mlsanity.com/paper/2402.18844/) (2024)
- [Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects](https://mlsanity.com/paper/2202.03274/) (2022)
- [HHP-Net: A light Heteroscedastic neural network for Head Pose estimation with uncertainty](https://mlsanity.com/paper/2111.01440/) (2021)
- [ActionPrompt: Action-Guided 3D Human Pose Estimation With Text and Pose Prompting](https://mlsanity.com/paper/2307.09026/) (2023)
- [CogStream: Context-guided Streaming Video Question Answering](https://mlsanity.com/paper/2506.10516/) (2025)
- [H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers](https://mlsanity.com/paper/2509.06956/) (2025)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [PedRecNet: Multi-task deep neural network for full 3D human pose and orientation estimation](https://mlsanity.com/paper/2204.11548/) (2022)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [MMHOI: Modeling Complex 3D Multi-Human Multi-Object Interactions](https://mlsanity.com/paper/2510.07828/) (2025)
- [Learning Twofold Heterogeneous Multi-Task by Sharing Similar Convolution Kernel Pairs](https://mlsanity.com/paper/2101.12431/) (2021)
- [Mamba-Driven Topology Fusion for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2505.20611/) (2025)
- [UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perception](https://mlsanity.com/paper/2602.01594/) (2026)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [Multi-task deep learning for large-scale building detail extraction from high-resolution satellite imagery](https://mlsanity.com/paper/2310.18899/) (2023)
- [Multi-hop graph transformer network for 3D human pose estimation](https://mlsanity.com/paper/2405.03055/) (2024)
- [Multi-task Learning For Joint Action and Gesture Recognition](https://mlsanity.com/paper/2505.17867/) (2025)
- [Spectral Compression Transformer with Line Pose Graph for Monocular 3D Human Pose Estimation](https://mlsanity.com/paper/2505.21309/) (2025)
- [Decoding Children's Gait Behavior](https://mlsanity.com/paper/2608.00371/) (2026)
- [Can Super Resolution be used to improve Human Pose Estimation in Low Resolution Scenarios?](https://mlsanity.com/paper/2107.02108/) (2021)
- [Learnable Triangulation for Deep Learning-based 3D Reconstruction of Objects of Arbitrary Topology from Single RGB Images](https://mlsanity.com/paper/2109.11844/) (2021)

## Abstract (arXiv)

Human pose estimation and action recognition are related tasks since both problems are strongly dependent on the human body representation and analysis. Nonetheless, most recent methods in the literature handle the two problems separately. In this work, we propose a multi-task framework for jointly estimating 2D or 3D human poses from monocular color images and classifying human actions from video sequences. We show that a single architecture can be used to solve both problems in an efficient way and still achieves state-of-the-art or comparable results at each task while running at more than 100 frames per second. The proposed method benefits from high parameters sharing between the two tasks by unifying still images and video clips processing in a single pipeline, allowing the model to be trained with data from different categories simultaneously and in a seamlessly way. Additionally, we provide important insights for end-to-end training the proposed multi-task model by decoupling key prediction parts, which consistently leads to better accuracy on both tasks. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU RGB+D) demonstrate the effectiveness of our method on the targeted tasks. Our source code and trained weights are publicly available at https://github.com/dluvizon/deephar.