# Continual Spatio-Temporal Graph Convolutional Networks

arXiv: https://arxiv.org/abs/2203.11009 · HTML: https://mlsanity.com/paper/2203.11009/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1902.09130/) (2019)
- [Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition](https://mlsanity.com/paper/1607.07043/) (2016)
- [Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/1904.01189/) (2019)
- [View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1703.08274/) (2017)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks](https://mlsanity.com/paper/2008.07404/) (2020)
- [Interpretable 3D Human Action Analysis with Temporal Convolutional Networks](https://mlsanity.com/paper/1704.04516/) (2017)
- [Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching](https://mlsanity.com/paper/1911.04131/) (2019)
- [Space-Time Representation of People Based on 3D Skeletal Data: A Review](https://mlsanity.com/paper/1601.01006/) (2016)
- [Continual Transformers: Redundancy-Free Attention for Online Inference](https://mlsanity.com/paper/2201.06268/) (2022)
- [Progressive Spatio-Temporal Graph Convolutional Network for Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2011.05668/) (2020)
- [Continual 3D Convolutional Neural Networks for Real-time Processing of Videos](https://mlsanity.com/paper/2106.00050/) (2021)
- [Temporal Attention-Augmented Graph Convolutional Network for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2010.12221/) (2020)

## Built on by (most cited first, 8 of 8)

- [Continual Learning on Graphs: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2402.11565/) (2024)
- [Continual Graph Learning: A Survey](https://mlsanity.com/paper/2301.12230/) (2023)
- [Graph Learning under Distribution Shifts: A Comprehensive Survey on Domain Adaptation, Out-of-distribution, and Continual Learning](https://mlsanity.com/paper/2402.16374/) (2024)
- [Geometric Deep Learning for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2402.17695/) (2024)
- [Continual Low-Rank Scaled Dot-product Attention](https://mlsanity.com/paper/2412.03214/) (2024)
- [AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition](https://mlsanity.com/paper/2507.19840/) (2025)
- [DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams](https://mlsanity.com/paper/2511.17693/) (2025)
- [Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning](https://mlsanity.com/paper/2607.23077/) (2026)

## Abstract (arXiv)

Graph-based reasoning over skeleton data has emerged as a promising approach for human action recognition. However, the application of prior graph-based methods, which predominantly employ whole temporal sequences as their input, to the setting of online inference entails considerable computational redundancy. In this paper, we tackle this issue by reformulating the Spatio-Temporal Graph Convolutional Neural Network as a Continual Inference Network, which can perform step-by-step predictions in time without repeat frame processing. To evaluate our method, we create a continual version of ST-GCN, CoST-GCN, alongside two derived methods with different self-attention mechanisms, CoAGCN and CoS-TR. We investigate weight transfer strategies and architectural modifications for inference acceleration, and perform experiments on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets. Retaining similar predictive accuracy, we observe up to 109x reduction in time complexity, on-hardware accelerations of 26x, and reductions in maximum allocated memory of 52% during online inference.