# HiT-DVAE: Human Motion Generation via Hierarchical Transformer Dynamical VAE

arXiv: https://arxiv.org/abs/2204.01565 · HTML: https://mlsanity.com/paper/2204.01565/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 59 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [DLow: Diversifying Latent Flows for Diverse Human Motion Prediction](https://mlsanity.com/paper/2003.08386/) (2020)
- [HP-GAN: Probabilistic 3D human motion prediction via GAN](https://mlsanity.com/paper/1711.09561/) (2017)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [Disentangled Sequential Autoencoder](https://mlsanity.com/paper/1803.02991/) (2018)
- [Sequential Neural Models with Stochastic Layers](https://mlsanity.com/paper/1605.07571/) (2016)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [Human Motion Prediction via Spatio-Temporal Inpainting](https://mlsanity.com/paper/1812.05478/) (2018)
- [MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction](https://mlsanity.com/paper/2108.07152/) (2021)
- [Deep representation learning for human motion prediction and classification](https://mlsanity.com/paper/1702.07486/) (2017)
- [The Pose Knows: Video Forecasting by Generating Pose Futures](https://mlsanity.com/paper/1705.00053/) (2017)
- [MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics](https://mlsanity.com/paper/1808.04545/) (2018)

## Built on by (most cited first, 16 of 16)

- [BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction](https://mlsanity.com/paper/2211.14304/) (2022)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Space](https://mlsanity.com/paper/2207.07351/) (2022)
- [Dynamic Motion Blending for Versatile Motion Editing](https://mlsanity.com/paper/2503.20724/) (2025)
- [Omni-Supervised Motion Editing: Balancing Change and Invariance through Positive-Negative Learning](https://mlsanity.com/paper/2605.30969/) (2026)
- [Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World](https://mlsanity.com/paper/2511.00041/) (2025)
- [Autoregressive GAN for Semantic Unconditional Head Motion Generation](https://mlsanity.com/paper/2211.00987/) (2022)
- [Speech Modeling with a Hierarchical Transformer Dynamical VAE](https://mlsanity.com/paper/2303.09404/) (2023)
- [Motion-DVAE: Unsupervised learning for fast human motion denoising](https://mlsanity.com/paper/2306.05846/) (2023)
- [Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation](https://mlsanity.com/paper/2312.04167/) (2023)
- [Multi-granular body modeling with Redundancy-Free Spatiotemporal Fusion for Text-Driven Motion Generation](https://mlsanity.com/paper/2503.06897/) (2025)
- [EquiFusion: Kinematics-Agnostic Human Motion Prediction via Equivariant Latent Diffusion](https://mlsanity.com/paper/2607.10984/) (2026)
- [Spatial Temporal Synergy: Balancing Change and Invariance in Text Driven 3D Human Motion Editing](https://mlsanity.com/paper/2608.16008/) (2026)
- [Personalized Pose Forecasting](https://mlsanity.com/paper/2312.03528/) (2023)
- [Motion Capture is Not the Target Domain: Scaling Synthetic Data for Learning Motion Representations](https://mlsanity.com/paper/2602.11064/) (2026)
- [InterEdit: Navigating Text-Guided 3D Dyadic Human Motion Editing](https://mlsanity.com/paper/2603.13082/) (2026)

## Abstract (arXiv)

Studies on the automatic processing of 3D human pose data have flourished in the recent past. In this paper, we are interested in the generation of plausible and diverse future human poses following an observed 3D pose sequence. Current methods address this problem by injecting random variables from a single latent space into a deterministic motion prediction framework, which precludes the inherent multi-modality in human motion generation. In addition, previous works rarely explore the use of attention to select which frames are to be used to inform the generation process up to our knowledge. To overcome these limitations, we propose Hierarchical Transformer Dynamical Variational Autoencoder, HiT-DVAE, which implements auto-regressive generation with transformer-like attention mechanisms. HiT-DVAE simultaneously learns the evolution of data and latent space distribution with time correlated probabilistic dependencies, thus enabling the generative model to learn a more complex and time-varying latent space as well as diverse and realistic human motions. Furthermore, the auto-regressive generation brings more flexibility on observation and prediction, i.e. one can have any length of observation and predict arbitrary large sequences of poses with a single pre-trained model. We evaluate the proposed method on HumanEva-I and Human3.6M with various evaluation methods, and outperform the state-of-the-art methods on most of the metrics.