# Human Motion Prediction via Spatio-Temporal Inpainting

arXiv: https://arxiv.org/abs/1812.05478 · HTML: https://mlsanity.com/paper/1812.05478/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2018.

## Built on by (most cited first, 30 of 114)

- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras](https://mlsanity.com/paper/2112.01524/) (2021)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [COUCH: Towards Controllable Human-Chair Interactions](https://mlsanity.com/paper/2205.00541/) (2022)
- [Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders](https://mlsanity.com/paper/2108.06720/) (2021)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction](https://mlsanity.com/paper/2108.07152/) (2021)
- [Back to MLP: A Simple Baseline for Human Motion Prediction](https://mlsanity.com/paper/2207.01567/) (2022)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [Human Motion Generation: A Survey](https://mlsanity.com/paper/2307.10894/) (2023)
- [SAGA: Stochastic Whole-Body Grasping with Contact](https://mlsanity.com/paper/2112.10103/) (2021)
- [Space-Time-Separable Graph Convolutional Network for Pose Forecasting](https://mlsanity.com/paper/2110.04573/) (2021)
- [HumanMAC: Masked Motion Completion for Human Motion Prediction](https://mlsanity.com/paper/2302.03665/) (2023)
- [Multi-Person Extreme Motion Prediction](https://mlsanity.com/paper/2105.08825/) (2021)
- [Generating Smooth Pose Sequences for Diverse Human Motion Prediction](https://mlsanity.com/paper/2108.08422/) (2021)
- [Single Motion Diffusion](https://mlsanity.com/paper/2302.05905/) (2023)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [Single-Shot Motion Completion with Transformer](https://mlsanity.com/paper/2103.00776/) (2021)
- [MoDi: Unconditional Motion Synthesis from Diverse Data](https://mlsanity.com/paper/2206.08010/) (2022)
- [Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation](https://mlsanity.com/paper/2305.09662/) (2023)
- [ParCo: Part-Coordinating Text-to-Motion Synthesis](https://mlsanity.com/paper/2403.18512/) (2024)
- [ReGenNet: Towards Human Action-Reaction Synthesis](https://mlsanity.com/paper/2403.11882/) (2024)
- [Spatio-Temporal Gating-Adjacency GCN for Human Motion Prediction](https://mlsanity.com/paper/2203.01474/) (2022)
- [Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformers](https://mlsanity.com/paper/2109.07531/) (2021)
- [Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations](https://mlsanity.com/paper/2303.13129/) (2023)

## Abstract (arXiv)

We propose a Generative Adversarial Network (GAN) to forecast 3D human motion given a sequence of past 3D skeleton poses. While recent GANs have shown promising results, they can only forecast plausible motion over relatively short periods of time (few hundred milliseconds) and typically ignore the absolute position of the skeleton w.r.t. the camera. Our scheme provides long term predictions (two seconds or more) for both the body pose and its absolute position. Our approach builds upon three main contributions. First, we represent the data using a spatio-temporal tensor of 3D skeleton coordinates which allows formulating the prediction problem as an inpainting one, for which GANs work particularly well. Secondly, we design an architecture to learn the joint distribution of body poses and global motion, capable to hypothesize large chunks of the input 3D tensor with missing data. And finally, we argue that the L2 metric, considered so far by most approaches, fails to capture the actual distribution of long-term human motion. We propose two alternative metrics, based on the distribution of frequencies, that are able to capture more realistic motion patterns. Extensive experiments demonstrate our approach to significantly improve the state of the art, while also handling situations in which past observations are corrupted by occlusions, noise and missing frames.