# Learning Constrained Dynamic Correlations in Spatiotemporal Graphs for Motion Prediction

arXiv: https://arxiv.org/abs/2204.01297 · HTML: https://mlsanity.com/paper/2204.01297/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 65 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction](https://mlsanity.com/paper/2003.08802/) (2020)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Human Motion Prediction via Spatio-Temporal Inpainting](https://mlsanity.com/paper/1812.05478/) (2018)
- [PU-GCN: Point Cloud Upsampling using Graph Convolutional Networks](https://mlsanity.com/paper/1912.03264/) (2019)
- [MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction](https://mlsanity.com/paper/2108.07152/) (2021)
- [SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2104.01528/) (2021)
- [Space-Time-Separable Graph Convolutional Network for Pose Forecasting](https://mlsanity.com/paper/2110.04573/) (2021)
- [BiHMP-GAN: Bidirectional 3D Human Motion Prediction GAN](https://mlsanity.com/paper/1812.02591/) (2018)
- [Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction](https://mlsanity.com/paper/1909.13245/) (2019)
- [Modeling Human Motion with Quaternion-based Neural Networks](https://mlsanity.com/paper/1901.07677/) (2019)
- [Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2110.04076/) (2021)
- [Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamic](https://mlsanity.com/paper/1805.02513/) (2018)
- [Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction](https://mlsanity.com/paper/2108.11244/) (2021)
- [Human Motion Prediction via Learning Local Structure Representations and Temporal Dependencies](https://mlsanity.com/paper/1902.07367/) (2019)
- [Aggregated Multi-GANs for Controlled 3D Human Motion Prediction](https://mlsanity.com/paper/2103.09755/) (2021)
- [PVRED: A Position-Velocity Recurrent Encoder-Decoder for Human Motion Prediction](https://mlsanity.com/paper/1906.06514/) (2019)
- [Online Multi-Agent Forecasting with Interpretable Collaborative Graph Neural Network](https://mlsanity.com/paper/2107.00894/) (2021)
- [Motion Prediction via Joint Dependency Modeling in Phase Space](https://mlsanity.com/paper/2201.02365/) (2022)

## Built on by (most cited first, 7 of 7)

- [Context-based Interpretable Spatio-Temporal Graph Convolutional Network for Human Motion Forecasting](https://mlsanity.com/paper/2402.19237/) (2024)
- [Recent Advances in Deterministic Human Motion Prediction: A Review](https://mlsanity.com/paper/2312.06184/) (2023)
- [Fooling Neural Networks for Motion Forecasting via Adversarial Attacks](https://mlsanity.com/paper/2403.04954/) (2024)
- [Temporal and Heterogeneous Graph Neural Network for Remaining Useful Life Prediction](https://mlsanity.com/paper/2405.04336/) (2024)
- [UnityGraph: Unified Learning of Spatio-temporal features for Multi-person Motion Prediction](https://mlsanity.com/paper/2411.04151/) (2024)
- [Multi-Scale Incremental Modeling for Enhanced Human Motion Prediction in Human-Robot Collaboration](https://mlsanity.com/paper/2412.11632/) (2024)
- [Zero-Gated Language-conditioned Human Motion Prediction](https://mlsanity.com/paper/2606.29208/) (2026)

## Abstract (arXiv)

Human motion prediction is challenging due to the complex spatiotemporal feature modeling. Among all methods, graph convolution networks (GCNs) are extensively utilized because of their superiority in explicit connection modeling. Within a GCN, the graph correlation adjacency matrix drives feature aggregation and is the key to extracting predictive motion features. State-of-the-art methods decompose the spatiotemporal correlation into spatial correlations for each frame and temporal correlations for each joint. Directly parameterizing these correlations introduces redundant parameters to represent common relations shared by all frames and all joints. Besides, the spatiotemporal graph adjacency matrix is the same for different motion samples and cannot reflect sample-wise correspondence variances. To overcome these two bottlenecks, we propose dynamic spatiotemporal decompose GC (DSTD-GC), which only takes 28.6% parameters of the state-of-the-art GC. The key of DSTD-GC is constrained dynamic correlation modeling, which explicitly parameterizes the common static constraints as a spatial/temporal vanilla adjacency matrix shared by all frames/joints and dynamically extracts correspondence variances for each frame/joint with an adjustment modeling function. For each sample, the common constrained adjacency matrices are fixed to represent generic motion patterns, while the extracted variances complete the matrices with specific pattern adjustments. Meanwhile, we mathematically reformulate GCs on spatiotemporal graphs into a unified form and find that DSTD-GC relaxes certain constraints of other GC, which contributes to a better representation capability. By combining DSTD-GC with prior knowledge, we propose a powerful spatiotemporal GCN called DSTD-GCN, which outperforms SOTA methods by 3.9% ~ 8.7% in prediction accuracy with 55.0% ~ 96.9% fewer parameters.