# STAR: Sparse Transformer-based Action Recognition

arXiv: https://arxiv.org/abs/2107.07089 · HTML: https://mlsanity.com/paper/2107.07089/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition](https://mlsanity.com/paper/2010.09978/) (2020)
- [SimGNN: A Neural Network Approach to Fast Graph Similarity Computation](https://mlsanity.com/paper/1808.05689/) (2018)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Memory Attention Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1804.08254/) (2018)
- [Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1705.08106/) (2017)

## Built on by (most cited first, 8 of 8)

- [Vision Transformers for Action Recognition: A Survey](https://mlsanity.com/paper/2209.05700/) (2022)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [Delving Deep into One-Shot Skeleton-based Action Recognition with Diverse Occlusions](https://mlsanity.com/paper/2202.11423/) (2022)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [Human Pose-based Estimation, Tracking and Action Recognition with Deep Learning: A Survey](https://mlsanity.com/paper/2310.13039/) (2023)
- [3D Skeleton-Based Action Recognition: A Review](https://mlsanity.com/paper/2506.00915/) (2025)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)

## Abstract (arXiv)

The cognitive system for human action and behavior has evolved into a deep learning regime, and especially the advent of Graph Convolution Networks has transformed the field in recent years. However, previous works have mainly focused on over-parameterized and complex models based on dense graph convolution networks, resulting in low efficiency in training and inference. Meanwhile, the Transformer architecture-based model has not yet been well explored for cognitive application in human action and behavior estimation. This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data. Our model can also process the variable length of video clips grouped as a single batch. Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference. Experiments show that our model achieves 4~18x speedup and 1/7~1/15 model size compared with the baseline models at competitive accuracy.