# Multi-Scale Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition

arXiv: https://arxiv.org/abs/2206.13028 · HTML: https://mlsanity.com/paper/2206.13028/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 28 of 28)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Learning Convolutional Neural Networks for Graphs](https://mlsanity.com/paper/1605.05273/) (2016)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [Skeleton-based Action Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/1704.07595/) (2017)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction](https://mlsanity.com/paper/1910.02212/) (2019)
- [RGB-D-based Human Motion Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/1711.08362/) (2017)
- [Skeleton Image Representation for 3D Action Recognition based on Tree Structure and Reference Joints](https://mlsanity.com/paper/1909.05704/) (2019)

## Built on by (most cited first, 30 of 71)

- [Hierarchically Decomposed Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.10741/) (2022)
- [Learning Discriminative Representations for Skeleton Based Action Recognition](https://mlsanity.com/paper/2303.03729/) (2023)
- [SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition](https://mlsanity.com/paper/2403.09508/) (2024)
- [HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations](https://mlsanity.com/paper/2303.06242/) (2023)
- [Generative Action Description Prompts for Skeleton-based Action Recognition](https://mlsanity.com/paper/2208.05318/) (2022)
- [Hypergraph Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2211.09590/) (2022)
- [Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition](https://mlsanity.com/paper/2411.18941/) (2024)
- [Language Knowledge-Assisted Representation Learning for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2305.12398/) (2023)
- [Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences](https://mlsanity.com/paper/2302.09018/) (2023)
- [Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling](https://mlsanity.com/paper/2303.15270/) (2023)
- [Focal and Global Spatial-Temporal Transformer for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.02693/) (2022)
- [Graph Contrastive Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2301.10900/) (2023)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [PSUMNet: Unified Modality Part Streams are All You Need for Efficient Pose-based Action Recognition](https://mlsanity.com/paper/2208.05775/) (2022)
- [Leveraging Spatio-Temporal Dependency for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2212.04761/) (2022)
- [One-Shot Action Recognition via Multi-Scale Spatial-Temporal Skeleton Matching](https://mlsanity.com/paper/2307.07286/) (2023)
- [Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2407.15706/) (2024)
- [InfoGCN++: Learning Representation by Predicting the Future for Online Human Skeleton-based Action Recognition](https://mlsanity.com/paper/2310.10547/) (2023)
- [Motion Matters: Motion-guided Modulation Network for Skeleton-based Micro-Action Recognition](https://mlsanity.com/paper/2507.21977/) (2025)
- [Overcoming Topology Agnosticism: Enhancing Skeleton-Based Action Recognition through Redefined Skeletal Topology Awareness](https://mlsanity.com/paper/2305.11468/) (2023)
- [A Decoupled Spatio-Temporal Framework for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2312.05830/) (2023)
- [Explore Human Parsing Modality for Action Recognition](https://mlsanity.com/paper/2401.02138/) (2024)
- [Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization](https://mlsanity.com/paper/2306.07576/) (2023)
- [HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation](https://mlsanity.com/paper/2503.01175/) (2025)
- [FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation](https://mlsanity.com/paper/2306.11046/) (2023)
- [MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition](https://mlsanity.com/paper/2404.10210/) (2024)
- [Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition](https://mlsanity.com/paper/2407.12312/) (2024)
- [MoFM: A Large-Scale Human Motion Foundation Model](https://mlsanity.com/paper/2502.05432/) (2025)
- [Spatial Temporal Graph Attention Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2208.08599/) (2022)
- [Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton based Action Recognition](https://mlsanity.com/paper/2209.02986/) (2022)

## Abstract (arXiv)

Graph convolutional networks have been widely used for skeleton-based action recognition due to their excellent modeling ability of non-Euclidean data. As the graph convolution is a local operation, it can only utilize the short-range joint dependencies and short-term trajectory but fails to directly model the distant joints relations and long-range temporal information that are vital to distinguishing various actions. To solve this problem, we present a multi-scale spatial graph convolution (MS-GC) module and a multi-scale temporal graph convolution (MT-GC) module to enrich the receptive field of the model in spatial and temporal dimensions. Concretely, the MS-GC and MT-GC modules decompose the corresponding local graph convolution into a set of sub-graph convolution, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-graph convolutions, and each node could complete multiple spatial and temporal aggregations with its neighborhoods. The final equivalent receptive field is accordingly enlarged, which is capable of capturing both short- and long-range dependencies in spatial and temporal domains. By coupling these two modules as a basic block, we further propose a multi-scale spatial temporal graph convolutional network (MST-GCN), which stacks multiple blocks to learn effective motion representations for action recognition. The proposed MST-GCN achieves remarkable performance on three challenging benchmark datasets, NTU RGB+D, NTU-120 RGB+D and Kinetics-Skeleton, for skeleton-based action recognition.