# Pose And Joint-Aware Action Recognition

arXiv: https://arxiv.org/abs/2010.08164 · HTML: https://mlsanity.com/paper/2010.08164/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot](https://mlsanity.com/paper/2402.14654/) (2024)
- [On the Benefits of 3D Pose and Tracking for Human Action Recognition](https://mlsanity.com/paper/2304.01199/) (2023)
- [STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videos](https://mlsanity.com/paper/2301.00794/) (2023)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model](https://mlsanity.com/paper/2407.06162/) (2024)
- [WheelPoser: Sparse-IMU Based Body Pose Estimation for Wheelchair Users](https://mlsanity.com/paper/2409.08494/) (2024)
- [PLSM: A Parallelized Liquid State Machine for Unintentional Action Detection](https://mlsanity.com/paper/2105.09909/) (2021)
- [ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition](https://mlsanity.com/paper/2310.14416/) (2023)
- [PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognition](https://mlsanity.com/paper/2407.13750/) (2024)
- [Distillation of Human-Object Interaction Contexts for Action Recognition](https://mlsanity.com/paper/2112.09448/) (2021)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection](https://mlsanity.com/paper/2405.08204/) (2024)
- [ReL-SAR: Representation Learning for Skeleton Action Recognition with Convolutional Transformers and BYOL](https://mlsanity.com/paper/2409.05749/) (2024)
- [BigMaQ: A Big Macaque Motion and Animation Dataset Bridging Image and 3D Pose Representations](https://mlsanity.com/paper/2602.19874/) (2026)
- [JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts](https://mlsanity.com/paper/2412.13708/) (2024)

## Abstract (arXiv)

Recent progress on action recognition has mainly focused on RGB and optical flow features. In this paper, we approach the problem of joint-based action recognition. Unlike other modalities, constellation of joints and their motion generate models with succinct human motion information for activity recognition. We present a new model for joint-based action recognition, which first extracts motion features from each joint separately through a shared motion encoder before performing collective reasoning. Our joint selector module re-weights the joint information to select the most discriminative joints for the task. We also propose a novel joint-contrastive loss that pulls together groups of joint features which convey the same action. We strengthen the joint-based representations by using a geometry-aware data augmentation technique which jitters pose heatmaps while retaining the dynamics of the action. We show large improvements over the current state-of-the-art joint-based approaches on JHMDB, HMDB, Charades, AVA action recognition datasets. A late fusion with RGB and Flow-based approaches yields additional improvements. Our model also outperforms the existing baseline on Mimetics, a dataset with out-of-context actions.