# Human Action Recognition: Pose-based Attention draws focus to Hands

arXiv: https://arxiv.org/abs/1712.08002 · HTML: https://mlsanity.com/paper/1712.08002/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2017.

## Built on by (most cited first, 21 of 21)

- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living](https://mlsanity.com/paper/2105.08141/) (2021)
- [Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble](https://mlsanity.com/paper/2110.06161/) (2021)
- [A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding](https://mlsanity.com/paper/2204.04734/) (2022)
- [Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition](https://mlsanity.com/paper/2110.14994/) (2021)
- [Human Interaction Recognition Framework based on Interacting Body Part Attention](https://mlsanity.com/paper/2101.08967/) (2021)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [MV2MAE: Multi-View Video Masked Autoencoders](https://mlsanity.com/paper/2401.15900/) (2024)
- [ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization](https://mlsanity.com/paper/2208.09414/) (2022)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition](https://mlsanity.com/paper/2408.05421/) (2024)
- [Hands Off: A Handshake Interaction Detection and Localization Model for COVID-19 Threat Control](https://mlsanity.com/paper/2110.09571/) (2021)
- [Skeletal Human Action Recognition using Hybrid Attention based Graph Convolutional Network](https://mlsanity.com/paper/2207.05493/) (2022)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction](https://mlsanity.com/paper/2210.13605/) (2022)
- [Bengali Sign Language Recognition through Hand Pose Estimation using Multi-Branch Spatial-Temporal Attention Model](https://mlsanity.com/paper/2408.14111/) (2024)
- [Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers](https://mlsanity.com/paper/2306.09331/) (2023)
- [Learning Scene Flow With Skeleton Guidance For 3D Action Recognition](https://mlsanity.com/paper/2306.13285/) (2023)
- [MPT-PAR:Mix-Parameters Transformer for Panoramic Activity Recognition](https://mlsanity.com/paper/2408.00420/) (2024)

## Abstract (arXiv)

We propose a new spatio-temporal attention based mechanism for human action recognition able to automatically attend to the hands most involved into the studied action and detect the most discriminative moments in an action. Attention is handled in a recurrent manner employing Recurrent Neural Network (RNN) and is fully-differentiable. In contrast to standard soft-attention based mechanisms, our approach does not use the hidden RNN state as input to the attention model. Instead, attention distributions are extracted using external information: human articulated pose. We performed an extensive ablation study to show the strengths of this approach and we particularly studied the conditioning aspect of the attention mechanism. We evaluate the method on the largest currently available human action recognition dataset, NTU-RGB+D, and report state-of-the-art results. Other advantages of our model are certain aspects of explanability, as the spatial and temporal attention distributions at test time allow to study and verify on which parts of the input data the method focuses.