# Multi-Resolution Generative Modeling of Human Motion from Limited Data

arXiv: https://arxiv.org/abs/2411.16498 · HTML: https://mlsanity.com/paper/2411.16498/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 33)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [Structured Prediction Helps 3D Human Motion Modelling](https://mlsanity.com/paper/1910.09070/) (2019)
- [Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation](https://mlsanity.com/paper/2401.08559/) (2024)
- [Learning to Generate Diverse Dance Motions with Transformer](https://mlsanity.com/paper/2008.08171/) (2020)
- [Gesticulator: A framework for semantically-aware speech-driven gesture generation](https://mlsanity.com/paper/2001.09326/) (2020)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [Improved Techniques for Training Single-Image GANs](https://mlsanity.com/paper/2003.11512/) (2020)
- [Deep Regression on Manifolds: A 3D Rotation Case Study](https://mlsanity.com/paper/2103.16317/) (2021)
- [Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis](https://mlsanity.com/paper/2006.07364/) (2020)
- [Analyzing Input and Output Representations for Speech-Driven Gesture Generation](https://mlsanity.com/paper/1903.03369/) (2019)
- [MoGlow: Probabilistic and controllable motion synthesis using normalising flows](https://mlsanity.com/paper/1905.06598/) (2019)

## Abstract (arXiv)

We present a generative model that learns to synthesize human motion from limited training sequences. Our framework provides conditional generation and blending across multiple temporal resolutions. The model adeptly captures human motion patterns by integrating skeletal convolution layers and a multi-scale architecture. Our model contains a set of generative and adversarial networks, along with embedding modules, each tailored for generating motions at specific frame rates while exerting control over their content and details. Notably, our approach also extends to the synthesis of co-speech gestures, demonstrating its ability to generate synchronized gestures from speech inputs, even with limited paired data. Through direct synthesis of SMPL pose parameters, our approach avoids test-time adjustments to fit human body meshes. Experimental results showcase our model's ability to achieve extensive coverage of training examples, while generating diverse motions, as indicated by local and global diversity metrics.