# VersatileMotion: A Unified Framework for Motion Synthesis and Comprehension

arXiv: https://arxiv.org/abs/2411.17335 · HTML: https://mlsanity.com/paper/2411.17335/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2024.

## Built on by (most cited first, 9 of 9)

- [OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation](https://mlsanity.com/paper/2510.19789/) (2025)
- [PRISM: Streaming Human Motion Generation with Per-Joint Latent Decomposition](https://mlsanity.com/paper/2603.08590/) (2026)
- [UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars](https://mlsanity.com/paper/2605.14731/) (2026)
- [AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling](https://mlsanity.com/paper/2605.29488/) (2026)
- [SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling](https://mlsanity.com/paper/2608.24334/) (2026)
- [BiomechGPT: Extending Motion-Language Models to Clinical Motion Understanding](https://mlsanity.com/paper/2505.18465/) (2025)
- [Odoriko: A Shape-Aware Multimodal Diffusion Framework for Human Motion](https://mlsanity.com/paper/2606.21135/) (2026)
- [BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion](https://mlsanity.com/paper/2609.35407/) (2026)
- [CleanMDM: Clean Motion Diffusion Model for Multimodal Motion Cleanup](https://mlsanity.com/paper/2610.05411/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are, by design, inherently capable of multi-task learning: through a unified next-token prediction paradigm, they can naturally address a wide variety of downstream tasks. Prior work in the motion domain has demonstrated some generality by adapting LLMs via a Motion Tokenizer coupled with an autoregressive Transformer to generate and understand human motion. However, this generality remains limited in scope and yields only modest performance gains. We introduce VersatileMotion, a unified multimodal motion LLM that combines a novel motion tokenizer, integrating VQ-VAE with flow matching, and an autoregressive transformer backbone to seamlessly support at least nine distinct motion-related tasks. VersatileMotion is the first method to handle single-agent and multi-agent motions in a single framework and enable cross-modal conversion between motion, text, music, and speech, achieving state-of-the-art performance on seven of these tasks. Each sequence in MotionHub may include one or more of the following annotations: natural-language captions, music or audio clips, speech transcripts, and multi-agent interaction data. To facilitate evaluation, we define and release benchmark splits covering nine core tasks. Extensive experiments demonstrate the superior performance, versatility, and potential of VersatileMotion as a foundational model for future understanding and generation of motion.