# Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

arXiv: https://arxiv.org/abs/2202.02314 · HTML: https://mlsanity.com/paper/2202.02314/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 4 Feb 2022.

## Builds on (most cited first, 30 of 39)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition](https://mlsanity.com/paper/2010.09978/) (2020)
- [Long-term Temporal Convolutions for Action Recognition](https://mlsanity.com/paper/1604.04494/) (2016)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [Learning Skeletal Graph Neural Networks for Hard 3D Pose Estimation](https://mlsanity.com/paper/2108.07181/) (2021)
- [Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learning](https://mlsanity.com/paper/2108.01959/) (2021)
- [Going Deeper into Action Recognition: A Survey](https://mlsanity.com/paper/1605.04988/) (2016)
- [Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning](https://mlsanity.com/paper/1805.02335/) (2018)
- [Memory Attention Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1804.08254/) (2018)
- [Richly Activated Graph Convolutional Network for Robust Skeleton-based Action Recognition](https://mlsanity.com/paper/2008.03791/) (2020)
- [UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition](https://mlsanity.com/paper/2107.08580/) (2021)
- [Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2108.04536/) (2021)
- [Fusing Higher-order Features in Graph Neural Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/2105.01563/) (2021)
- [Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2007.03263/) (2020)
- [AdaSGN: Adapting Joint Number and Model Size for Efficient Skeleton-Based Action Recognition](https://mlsanity.com/paper/2103.11770/) (2021)

## Built on by (most cited first, 4 of 4)

- [Learning Discriminative Representations for Skeleton Based Action Recognition](https://mlsanity.com/paper/2303.03729/) (2023)
- [Temporal-Channel Topology Enhanced Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2302.12967/) (2023)
- [Towards Generalizing Temporal Action Segmentation to Unseen Views](https://mlsanity.com/paper/2504.02512/) (2025)
- [G3CN: Gaussian Topology Refinement Gated Graph Convolutional Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2509.07335/) (2025)

## Abstract (arXiv)

Graph Convolutional Networks (GCNs) have been widely used to model the high-order dynamic dependencies for skeleton-based action recognition. Most existing approaches do not explicitly embed the high-order spatio-temporal importance to joints' spatial connection topology and intensity, and they do not have direct objectives on their attention module to jointly learn when and where to focus on in the action sequence. To address these problems, we propose the To-a-T Spatio-Temporal Focus (STF), a skeleton-based action recognition framework that utilizes the spatio-temporal gradient to focus on relevant spatio-temporal features. We first propose the STF modules with learnable gradient-enforced and instance-dependent adjacency matrices to model the high-order spatio-temporal dynamics. Second, we propose three loss terms defined on the gradient-based spatio-temporal focus to explicitly guide the classifier when and where to look at, distinguish confusing classes, and optimize the stacked STF modules. STF outperforms the state-of-the-art methods on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets in all 15 settings over different views, subjects, setups, and input modalities, and STF also shows better accuracy on scarce data and dataset shifting settings.