# Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling

arXiv: https://arxiv.org/abs/1908.07214 · HTML: https://mlsanity.com/paper/1908.07214/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2019.

## Built on by (most cited first, 30 of 32)

- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Learning Motion Priors for 4D Human Body Capture in 3D Scenes](https://mlsanity.com/paper/2108.10399/) (2021)
- [Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure](https://mlsanity.com/paper/2111.12159/) (2021)
- [Multiscale Spatio-Temporal Graph Neural Networks for 3D Skeleton-Based Motion Prediction](https://mlsanity.com/paper/2108.11244/) (2021)
- [Real-time Controllable Motion Transition for Characters](https://mlsanity.com/paper/2205.02540/) (2022)
- [BASAR:Black-box Attack on Skeletal Action Recognition](https://mlsanity.com/paper/2103.05266/) (2021)
- [Conditional Motion In-betweening](https://mlsanity.com/paper/2202.04307/) (2022)
- [Motion In-Betweening with Phase Manifolds](https://mlsanity.com/paper/2308.12751/) (2023)
- [Understanding the Robustness of Skeleton-based Action Recognition under Adversarial Attack](https://mlsanity.com/paper/2103.05347/) (2021)
- [RSMT: Real-time Stylized Motion Transition for Characters](https://mlsanity.com/paper/2306.11970/) (2023)
- [GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction](https://mlsanity.com/paper/2110.00380/) (2021)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient](https://mlsanity.com/paper/2308.05681/) (2023)
- [Defending Black-box Skeleton-based Human Activity Classifiers](https://mlsanity.com/paper/2203.04713/) (2022)
- [Understanding the Vulnerability of Skeleton-based Human Activity Recognition via Black-box Attack](https://mlsanity.com/paper/2211.11312/) (2022)
- [Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding](https://mlsanity.com/paper/2208.00774/) (2022)
- [Focalized Contrastive View-invariant Learning for Self-supervised Skeleton-based Action Recognition](https://mlsanity.com/paper/2304.00858/) (2023)
- [Human Motion Prediction under Unexpected Perturbation](https://mlsanity.com/paper/2403.15891/) (2024)
- [Two-Person Interaction Augmentation with Skeleton Priors](https://mlsanity.com/paper/2404.05490/) (2024)
- [Pose Representations for Deep Skeletal Animation](https://mlsanity.com/paper/2111.13907/) (2021)
- [Decoupling Contact for Fine-Grained Motion Style Transfer](https://mlsanity.com/paper/2409.05387/) (2024)
- [NEURAL MARIONETTE: A Transformer-based Multi-action Human Motion Synthesis System](https://mlsanity.com/paper/2209.13204/) (2022)
- [Learning Joint Representation of Human Motion and Language](https://mlsanity.com/paper/2210.15187/) (2022)
- [STGFormer: Spatio-Temporal GraphFormer for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2407.10099/) (2024)
- [Towards Synthesized and Editable Motion In-Betweening Through Part-Wise Phase Representation](https://mlsanity.com/paper/2503.08180/) (2025)
- [Diverse Dance Synthesis via Keyframes with Transformer Controllers](https://mlsanity.com/paper/2207.05906/) (2022)
- [Motion Generation Review: Exploring Deep Learning for Lifelike Animation with Manifold](https://mlsanity.com/paper/2412.10458/) (2024)
- [Temporal-Spatial Processing of Event Camera Data via Delay-Loop Reservoir Neural Network](https://mlsanity.com/paper/2403.17013/) (2024)
- [FunPhase: A Periodic Functional Autoencoder for Motion Generation via Phase Manifolds](https://mlsanity.com/paper/2512.09423/) (2025)

## Abstract (arXiv)

Data-driven modeling of human motions is ubiquitous in computer graphics and computer vision applications, such as synthesizing realistic motions or recognizing actions. Recent research has shown that such problems can be approached by learning a natural motion manifold using deep learning to address the shortcomings of traditional data-driven approaches. However, previous methods can be sub-optimal for two reasons. First, the skeletal information has not been fully utilized for feature extraction. Unlike images, it is difficult to define spatial proximity in skeletal motions in the way that deep networks can be applied. Second, motion is time-series data with strong multi-modal temporal correlations. A frame could be followed by several candidate frames leading to different motions; long-range dependencies exist where a number of frames in the beginning correlate to a number of frames later. Ineffective modeling would either under-estimate the multi-modality and variance, resulting in featureless mean motion or over-estimate them resulting in jittery motions. In this paper, we propose a new deep network to tackle these challenges by creating a natural motion manifold that is versatile for many applications. The network has a new spatial component for feature extraction. It is also equipped with a new batch prediction model that predicts a large number of frames at once, such that long-term temporally-based objective functions can be employed to correctly learn the motion multi-modality and variances. With our system, long-duration motions can be predicted/synthesized using an open-loop setup where the motion retains the dynamics accurately. It can also be used for denoising corrupted motions and synthesizing new motions with given control signals. We demonstrate that our system can create superior results comparing to existing work in multiple applications.