# Temporal Consistency Two-Stream CNN for Human Motion Prediction

arXiv: https://arxiv.org/abs/2104.05015 · HTML: https://mlsanity.com/paper/2104.05015/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 11 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Skeleton-Based Action Recognition with Multi-Stream Adaptive Graph Convolutional Networks](https://mlsanity.com/paper/1912.06971/) (2019)
- [A Neural Temporal Model for Human Motion Prediction](https://mlsanity.com/paper/1809.03036/) (2018)
- [Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction](https://mlsanity.com/paper/1909.13245/) (2019)
- [Predicting Scene Parsing and Motion Dynamics in the Future](https://mlsanity.com/paper/1711.03270/) (2017)
- [Human Motion Prediction via Pattern Completion in Latent Representation Space](https://mlsanity.com/paper/1904.09039/) (2019)
- [Two-stream Fusion Model for Dynamic Hand Gesture Recognition using 3D-CNN and 2D-CNN Optical Flow guided Motion Template](https://mlsanity.com/paper/2007.08847/) (2020)

## Built on by (most cited first, 2 of 2)

- [Physics-constrained Attack against Convolution-based Human Motion Prediction](https://mlsanity.com/paper/2306.11990/) (2023)
- [Recent Advances in Deterministic Human Motion Prediction: A Review](https://mlsanity.com/paper/2312.06184/) (2023)

## Abstract (arXiv)

Fusion is critical for a two-stream network. In this paper, we propose a novel temporal fusion (TF) module to fuse the two-stream joints' information to predict human motion, including a temporal concatenation and a reinforcement trajectory spatial-temporal (TST) block, specifically designed to keep prediction temporal consistency. In particular, the temporal concatenation keeps the temporal consistency of preliminary predictions from two streams. Meanwhile, the TST block improves the spatial-temporal feature coupling. However, the TF module can increase the temporal continuities between the first predicted pose and the given poses and between each predicted pose. The fusion is based on a two-stream network that consists of a dynamic velocity stream (V-Stream) and a static position stream (P-Stream) because we found that the joints' velocity information improves the short-term prediction, while the joints' position information is better at long-term prediction, and they are complementary in motion prediction. Finally, our approach achieves impressive results on three benchmark datasets, including H3.6M, CMU-Mocap, and 3DPW in both short-term and long-term predictions, confirming its effectiveness and efficiency.