# Spatial Temporal Transformer Network for Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2012.06399 · HTML: https://mlsanity.com/paper/2012.06399/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 30 of 77)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation](https://mlsanity.com/paper/2108.07181/) (2021)
- [Generative Action Description Prompts for Skeleton-based Action Recognition](https://mlsanity.com/paper/2208.05318/) (2022)
- [Delving into Sequential Patches for Deepfake Detection](https://mlsanity.com/paper/2207.02803/) (2022)
- [GPGait: Generalized Pose-based Gait Recognition](https://mlsanity.com/paper/2303.05234/) (2023)
- [CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Prediction](https://mlsanity.com/paper/2203.04570/) (2022)
- [Hypergraph Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2211.09590/) (2022)
- [Spatial Transformer Network on Skeleton-based Gait Recognition](https://mlsanity.com/paper/2204.03873/) (2022)
- [Global-local Motion Transformer for Unsupervised Skeleton-based Action Learning](https://mlsanity.com/paper/2207.06101/) (2022)
- [DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition](https://mlsanity.com/paper/2203.10233/) (2022)
- [Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning](https://mlsanity.com/paper/2207.09644/) (2022)
- [SkeletonMAE: Spatial-Temporal Masked Autoencoders for Self-supervised Skeleton Action Recognition](https://mlsanity.com/paper/2209.02399/) (2022)
- [TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration](https://mlsanity.com/paper/2203.00927/) (2022)
- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)
- [Part Aware Contrastive Learning for Self-Supervised Action Recognition](https://mlsanity.com/paper/2305.00666/) (2023)
- [Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers](https://mlsanity.com/paper/2206.06131/) (2022)
- [GaitMixer: Skeleton-based Gait Representation Learning via Wide-spectrum Multi-axial Mixer](https://mlsanity.com/paper/2210.15491/) (2022)
- [Navigating Open Set Scenarios for Skeleton-based Action Recognition](https://mlsanity.com/paper/2312.06330/) (2023)
- [On the Utility of 3D Hand Poses for Action Recognition](https://mlsanity.com/paper/2403.09805/) (2024)
- [Delving Deep into One-Shot Skeleton-based Action Recognition with Diverse Occlusions](https://mlsanity.com/paper/2202.11423/) (2022)
- [Simba: Mamba augmented U-ShiftGCN for Skeletal Action Recognition in Videos](https://mlsanity.com/paper/2404.07645/) (2024)
- [Hierarchical Graph Convolutional Skeleton Transformer for Action Recognition](https://mlsanity.com/paper/2109.02860/) (2021)
- [Multimodal Transformer for Nursing Activity Recognition](https://mlsanity.com/paper/2204.04564/) (2022)
- [Pyramid Self-attention Polymerization Learning for Semi-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02327/) (2023)
- [GaitPT: Skeletons Are All You Need For Gait Recognition](https://mlsanity.com/paper/2308.10623/) (2023)
- [CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner](https://mlsanity.com/paper/2403.10082/) (2024)
- [Adaptive Hyper-Graph Convolution Network for Skeleton-based Human Action Recognition with Virtual Connections](https://mlsanity.com/paper/2411.14796/) (2024)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [Unsupervised Domain Adaptation for Video Transformers in Action Recognition](https://mlsanity.com/paper/2207.12842/) (2022)

## Abstract (arXiv)

Skeleton-based human action recognition has achieved a great interest in recent years, as skeleton data has been demonstrated to be robust to illumination changes, body scales, dynamic camera views, and complex background. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network which outperforms state-of-the-art models using the same input data on both NTU-RGB+D 60 and NTU-RGB+D 120.