# A structured latent space for human body motion generation

arXiv: https://arxiv.org/abs/2106.04387 · HTML: https://mlsanity.com/paper/2106.04387/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Real-time Deep Dynamic Characters](https://mlsanity.com/paper/2105.01794/) (2021)
- [Unsupervised Shape and Pose Disentanglement for 3D Meshes](https://mlsanity.com/paper/2007.11341/) (2020)
- [LIMP: Learning Latent Shape Representations with Metric Preservation Priors](https://mlsanity.com/paper/2003.12283/) (2020)
- [Predicting 3D Human Dynamics from Video](https://mlsanity.com/paper/1908.04781/) (2019)
- [Building Statistical Shape Spaces for 3D Human Modeling](https://mlsanity.com/paper/1503.05860/) (2015)
- [Learning Compositional Representation for 4D Captures with Neural ODE](https://mlsanity.com/paper/2103.08271/) (2021)
- [Disentangled Human Body Embedding Based on Deep Hierarchical Neural Network](https://mlsanity.com/paper/1905.05622/) (2019)
- [SoftSMPL: Data-driven Modeling of Nonlinear Soft-tissue Dynamics for Parametric Humans](https://mlsanity.com/paper/2004.00326/) (2020)
- [Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model](https://mlsanity.com/paper/2010.09950/) (2020)
- [Recovering Trajectories of Unmarked Joints in 3D Human Actions Using Latent Space Optimization](https://mlsanity.com/paper/2012.02043/) (2020)

## Built on by (most cited first, 6 of 6)

- [4DHumanOutfit: a multi-subject 4D dataset of human motion sequences in varying outfits exhibiting large displacements](https://mlsanity.com/paper/2306.07399/) (2023)
- [Correspondence-free online human motion retargeting](https://mlsanity.com/paper/2302.00556/) (2023)
- [Representing motion as a sequence of latent primitives, a flexible approach for human motion modelling](https://mlsanity.com/paper/2206.13142/) (2022)
- [Deformation-Guided Unsupervised Non-Rigid Shape Matching](https://mlsanity.com/paper/2311.15668/) (2023)
- [Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos](https://mlsanity.com/paper/2311.11662/) (2023)
- [Cepheid: Decoupling Scale from Structure for Cross-Family Motion Completion](https://mlsanity.com/paper/2610.18871/) (2026)

## Abstract (arXiv)

We propose a framework to learn a structured latent space to represent 4D human body motion, where each latent vector encodes a full motion of the whole 3D human shape. On one hand several data-driven skeletal animation models exist proposing motion spaces of temporally dense motion signals, but based on geometrically sparse kinematic representations. On the other hand many methods exist to build shape spaces of dense 3D geometry, but for static frames. We bring together both concepts, proposing a motion space that is dense both temporally and geometrically. Once trained, our model generates a multi-frame sequence of dense 3D meshes based on a single point in a low-dimensional latent space. This latent space is built to be structured, such that similar motions form clusters. It also embeds variations of duration in the latent vector, allowing semantically close sequences that differ only by temporal unfolding to share similar latent vectors. We demonstrate experimentally the structural properties of our latent space, and show it can be used to generate plausible interpolations between different actions. We also apply our model to 4D human motion completion, showing its promising abilities to learn spatio-temporal features of human motion.