# Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction

arXiv: https://arxiv.org/abs/1909.13245 · HTML: https://mlsanity.com/paper/1909.13245/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2019.

## Built on by (most cited first, 30 of 47)

- [CIRCLE: Capture In Rich Contextual Environments](https://mlsanity.com/paper/2303.17912/) (2023)
- [Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Prediction](https://mlsanity.com/paper/2203.16051/) (2022)
- [Camouflaged Object Detection via Context-aware Cross-level Fusion](https://mlsanity.com/paper/2207.13362/) (2022)
- [WANDR: Intention-guided Human Motion Generation](https://mlsanity.com/paper/2404.15383/) (2024)
- [MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition](https://mlsanity.com/paper/2405.02077/) (2024)
- [Multimodal Pre-training Based on Graph Attention Network for Document Understanding](https://mlsanity.com/paper/2203.13530/) (2022)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Relation-aware Compositional Zero-shot Learning for Attribute-Object Pair Recognition](https://mlsanity.com/paper/2108.04603/) (2021)
- [Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semi-Supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02316/) (2023)
- [Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks](https://mlsanity.com/paper/2404.08347/) (2024)
- [STGlow: A Flow-based Generative Framework with Dual Graphormer for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2211.11220/) (2022)
- [Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution](https://mlsanity.com/paper/2109.01664/) (2021)
- [Spatial-Temporal Federated Learning for Lifelong Person Re-identification on Distributed Edges](https://mlsanity.com/paper/2207.11759/) (2022)
- [Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration](https://mlsanity.com/paper/2212.07592/) (2022)
- [Pyramid Self-attention Polymerization Learning for Semi-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.02327/) (2023)
- [Dynamic Dense Graph Convolutional Network for Skeleton-based Human Motion Prediction](https://mlsanity.com/paper/2311.17408/) (2023)
- [Dyadic Human Motion Prediction](https://mlsanity.com/paper/2112.00396/) (2021)
- [Wavelet-Attention CNN for Image Classification](https://mlsanity.com/paper/2201.09271/) (2022)
- [GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition](https://mlsanity.com/paper/2401.10039/) (2024)
- [Learning Constrained Dynamic Correlations in Spatiotemporal Graphs for Motion Prediction](https://mlsanity.com/paper/2204.01297/) (2022)
- [EA-Net: Edge-Aware Network for Flow-based Video Frame Interpolation](https://mlsanity.com/paper/2105.07673/) (2021)
- [FDA-GAN: Flow-based Dual Attention GAN for Human Pose Transfer](https://mlsanity.com/paper/2112.00281/) (2021)
- [FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation](https://mlsanity.com/paper/2306.11046/) (2023)
- [Spatio-temporal Decoupled Knowledge Compensator for Few-Shot Action Recognition](https://mlsanity.com/paper/2602.18043/) (2026)
- [PV-RCNN++: Semantical Point-Voxel Feature Interaction for 3D Object Detection](https://mlsanity.com/paper/2208.13414/) (2022)
- [SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories](https://mlsanity.com/paper/2503.22869/) (2025)
- [Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning](https://mlsanity.com/paper/2508.10897/) (2025)
- [Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling](https://mlsanity.com/paper/2510.12385/) (2025)
- [Temporal Consistency Two-Stream CNN for Human Motion Prediction](https://mlsanity.com/paper/2104.05015/) (2021)
- [Modeling Regime Shifts in Multiple Time Series](https://mlsanity.com/paper/2109.09692/) (2021)

## Abstract (arXiv)

Human motion prediction aims to generate future motions based on the observed human motions. Witnessing the success of Recurrent Neural Networks (RNN) in modeling the sequential data, recent works utilize RNN to model human-skeleton motion on the observed motion sequence and predict future human motions. However, these methods did not consider the existence of the spatial coherence among joints and the temporal evolution among skeletons, which reflects the crucial characteristics of human motion in spatiotemporal space. To this end, we propose a novel Skeleton-joint Co-attention Recurrent Neural Networks (SC-RNN) to capture the spatial coherence among joints, and the temporal evolution among skeletons simultaneously on a skeleton-joint co-attention feature map in spatiotemporal space. First, a skeleton-joint feature map is constructed as the representation of the observed motion sequence. Second, we design a new Skeleton-joint Co-Attention (SCA) mechanism to dynamically learn a skeleton-joint co-attention feature map of this skeleton-joint feature map, which can refine the useful observed motion information to predict one future motion. Third, a variant of GRU embedded with SCA collaboratively models the human-skeleton motion and human-joint motion in spatiotemporal space by regarding the skeleton-joint co-attention feature map as the motion context. Experimental results on human motion prediction demonstrate the proposed method outperforms the related methods.