# Comparison of Spatio-Temporal Models for Human Motion and Pose Forecasting in Face-to-Face Interaction Scenarios

arXiv: https://arxiv.org/abs/2203.03245 · HTML: https://mlsanity.com/paper/2203.03245/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [3D Human Pose Estimation with Spatial and Temporal Transformers](https://mlsanity.com/paper/2103.10455/) (2021)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Context-aware Human Motion Prediction](https://mlsanity.com/paper/1904.03419/) (2019)
- [Learning Human Motion Models for Long-term Predictions](https://mlsanity.com/paper/1704.02827/) (2017)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild](https://mlsanity.com/paper/2104.04029/) (2021)
- [Socially and Contextually Aware Human Motion and Pose Forecasting](https://mlsanity.com/paper/2007.06843/) (2020)
- [Symbiotic Graph Neural Networks for 3D Skeleton-based Human Action Recognition and Motion Prediction](https://mlsanity.com/paper/1910.02212/) (2019)
- [Multi-level Motion Attention for Human Motion Prediction](https://mlsanity.com/paper/2106.09300/) (2021)
- [Context-Aware Personality Inference in Dyadic Scenarios: Introducing the UDIVA Dataset](https://mlsanity.com/paper/2012.14259/) (2020)
- [To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations](https://mlsanity.com/paper/1910.02181/) (2019)
- [Looking Ahead: Anticipating Pedestrians Crossing with Future Frames Prediction](https://mlsanity.com/paper/1910.09077/) (2019)
- [Simple Baseline for Single Human Motion Forecasting](https://mlsanity.com/paper/2110.07495/) (2021)
- [Forecasting Characteristic 3D Poses of Human Actions](https://mlsanity.com/paper/2011.15079/) (2020)
- [TrajeVAE: Controllable Human Motion Generation from Trajectories](https://mlsanity.com/paper/2104.00351/) (2021)
- [Didn't see that coming: a survey on non-verbal social human behavior forecasting](https://mlsanity.com/paper/2203.02480/) (2022)
- [Dyadic Human Motion Prediction](https://mlsanity.com/paper/2112.00396/) (2021)

## Built on by (most cited first, 8 of 8)

- [BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction](https://mlsanity.com/paper/2211.14304/) (2022)
- [Seamless Human Motion Composition with Blended Positional Encodings](https://mlsanity.com/paper/2402.15509/) (2024)
- [in2IN: Leveraging individual Information to Generate Human INteractions](https://mlsanity.com/paper/2404.09988/) (2024)
- [REACT 2024: the Second Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2401.05166/) (2024)
- [Nonisotropic Gaussian Diffusion for Realistic 3D Human Motion Prediction](https://mlsanity.com/paper/2501.06035/) (2025)
- [From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models](https://mlsanity.com/paper/2504.05265/) (2025)
- [REACT2023: the first Multi-modal Multiple Appropriate Facial Reaction Generation Challenge](https://mlsanity.com/paper/2306.06583/) (2023)
- [Latent Behavior Diffusion for Sequential Reaction Generation in Dyadic Setting](https://mlsanity.com/paper/2505.07901/) (2025)

## Abstract (arXiv)

Human behavior forecasting during human-human interactions is of utmost importance to provide robotic or virtual agents with social intelligence. This problem is especially challenging for scenarios that are highly driven by interpersonal dynamics. In this work, we present the first systematic comparison of state-of-the-art approaches for behavior forecasting. To do so, we leverage whole-body annotations (face, body, and hands) from the very recently released UDIVA v0.5, which features face-to-face dyadic interactions. Our best attention-based approaches achieve state-of-the-art performance in UDIVA v0.5. We show that by autoregressively predicting the future with methods trained for the short-term future (<400ms), we outperform the baselines even for a considerably longer-term future (up to 2s). We also show that this finding holds when highly noisy annotations are used, which opens new horizons towards the use of weakly-supervised learning. Combined with large-scale datasets, this may help boost the advances in this field.