# Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2108.04536 · HTML: https://mlsanity.com/paper/2108.04536/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks](https://mlsanity.com/paper/1603.07772/) (2016)
- [Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning](https://mlsanity.com/paper/1805.02335/) (2018)
- [SkeleMotion: A New Representation of Skeleton Joint Sequences Based on Motion Information for 3D Action Recognition](https://mlsanity.com/paper/1907.13025/) (2019)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [Skeleton Image Representation for 3D Action Recognition based on Tree Structure and Reference Joints](https://mlsanity.com/paper/1909.05704/) (2019)
- [Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2011.13322/) (2020)
- [Multi Scale Temporal Graph Networks For Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.02970/) (2020)

## Built on by (most cited first, 25 of 25)

- [STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition](https://mlsanity.com/paper/2210.07503/) (2022)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Uplift and Upsample: Efficient 3D Human Pose Estimation with Uplifting Transformers](https://mlsanity.com/paper/2210.06110/) (2022)
- [Cross-Modal Learning with 3D Deformable Attention for Action Recognition](https://mlsanity.com/paper/2212.05638/) (2022)
- [Unified Keypoint-based Action Recognition Framework via Structured Keypoint Pooling](https://mlsanity.com/paper/2303.15270/) (2023)
- [IIP-Transformer: Intra-Inter-Part Transformer for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2110.13385/) (2021)
- [PSUMNet: Unified Modality Part Streams are All You Need for Efficient Pose-based Action Recognition](https://mlsanity.com/paper/2208.05775/) (2022)
- [Skeleton-based Action Recognition via Temporal-Channel Aggregation](https://mlsanity.com/paper/2205.15936/) (2022)
- [Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2407.15706/) (2024)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition](https://mlsanity.com/paper/2402.02210/) (2024)
- [Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2202.02314/) (2022)
- [Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton based Action Recognition](https://mlsanity.com/paper/2209.02986/) (2022)
- [Towards Activated Muscle Group Estimation in the Wild](https://mlsanity.com/paper/2303.00952/) (2023)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [Exploring Time Granularity on Temporal Graphs for Dynamic Link Prediction in Real-world Networks](https://mlsanity.com/paper/2311.12255/) (2023)
- [LSTA-Net: Long short-term Spatio-Temporal Aggregation Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2111.00823/) (2021)
- [Self-attention based anchor proposal for skeleton-based action recognition](https://mlsanity.com/paper/2112.09413/) (2021)
- [Part-aware Prototypical Graph Network for One-shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2208.09150/) (2022)
- [Pose-Guided Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2210.06192/) (2022)
- [SLGTformer: An Attention-Based Approach to Sign Language Recognition](https://mlsanity.com/paper/2212.10746/) (2022)
- [SkeleTR: Towrads Skeleton-based Action Recognition in the Wild](https://mlsanity.com/paper/2309.11445/) (2023)
- [Cefdet: Cognitive Effectiveness Network Based on Fuzzy Inference for Action Detection](https://mlsanity.com/paper/2410.05771/) (2024)
- [HFGCN:Hypergraph Fusion Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2501.11007/) (2025)
- [NeuroPath: Brain-Inspired Dual-Pathway Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2608.17487/) (2026)

## Abstract (arXiv)

The task of skeleton-based action recognition remains a core challenge in human-centred scene understanding due to the multiple granularities and large variation in human motion. Existing approaches typically employ a single neural representation for different motion patterns, which has difficulty in capturing fine-grained action classes given limited training data. To address the aforementioned problems, we propose a novel multi-granular spatio-temporal graph network for skeleton-based action classification that jointly models the coarse- and fine-grained skeleton motion patterns. To this end, we develop a dual-head graph network consisting of two interleaved branches, which enables us to extract features at two spatio-temporal resolutions in an effective and efficient manner. Moreover, our network utilises a cross-head communication strategy to mutually enhance the representations of both heads. We conducted extensive experiments on three large-scale datasets, namely NTU RGB+D 60, NTU RGB+D 120, and Kinetics-Skeleton, and achieves the state-of-the-art performance on all the benchmarks, which validates the effectiveness of our method.