# Large Motion Model for Unified Multi-Modal Motion Generation

arXiv: https://arxiv.org/abs/2404.01284 · HTML: https://mlsanity.com/paper/2404.01284/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 146 of its 172 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 30 of 146)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)

## Built on by (most cited first, 30 of 65)

- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data](https://mlsanity.com/paper/2507.07095/) (2025)
- [GENMO: A GENeralist Model for Human MOtion](https://mlsanity.com/paper/2505.01425/) (2025)
- [The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion](https://mlsanity.com/paper/2412.10523/) (2024)
- [Puppeteer: Rig and Animate Your 3D Models](https://mlsanity.com/paper/2508.10898/) (2025)
- [Motion Anything: Any to Motion Generation](https://mlsanity.com/paper/2503.06955/) (2025)
- [Vivid-ZOO: Multi-View Video Generation with Diffusion Model](https://mlsanity.com/paper/2406.08659/) (2024)
- [M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation](https://mlsanity.com/paper/2405.16273/) (2024)
- [MotionGPT3: Human Motion as a Second Modality](https://mlsanity.com/paper/2506.24086/) (2025)
- [Kimodo: Scaling Controllable Human Motion Generation](https://mlsanity.com/paper/2603.15546/) (2026)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm](https://mlsanity.com/paper/2502.02358/) (2025)
- [EgoLM: Multi-Modal Language Model of Egocentric Motions](https://mlsanity.com/paper/2409.18127/) (2024)
- [SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction](https://mlsanity.com/paper/2503.18211/) (2025)
- [MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls](https://mlsanity.com/paper/2407.21136/) (2024)
- [AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation](https://mlsanity.com/paper/2410.07164/) (2024)
- [Scaling Large Motion Models with Million-Level Human Motions](https://mlsanity.com/paper/2410.03311/) (2024)
- [Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns](https://mlsanity.com/paper/2410.20389/) (2024)
- [Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing](https://mlsanity.com/paper/2410.18977/) (2024)
- [SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization](https://mlsanity.com/paper/2412.05095/) (2024)
- [MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations](https://mlsanity.com/paper/2410.13790/) (2024)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [Driving with Regulation: Trustworthy and Interpretable Decision-Making for Autonomous Driving with Retrieval-Augmented Reasoning](https://mlsanity.com/paper/2410.04759/) (2024)
- [MixerMDM: Learnable Composition of Human Motion Diffusion Models](https://mlsanity.com/paper/2504.01019/) (2025)
- [ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment](https://mlsanity.com/paper/2511.19217/) (2025)
- [MOSPA: Human Motion Generation Driven by Spatial Audio](https://mlsanity.com/paper/2507.11949/) (2025)
- [Foundation Model for Skeleton-Based Human Action Understanding](https://mlsanity.com/paper/2508.12586/) (2025)
- [Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model](https://mlsanity.com/paper/2508.07863/) (2025)
- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)
- [HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding](https://mlsanity.com/paper/2503.12955/) (2025)

## Abstract (arXiv)

Human motion generation, a cornerstone technique in animation and video production, has widespread applications in various tasks like text-to-motion and music-to-dance. Previous works focus on developing specialist models tailored for each task without scalability. In this work, we present Large Motion Model (LMM), a motion-centric, multi-modal framework that unifies mainstream motion generation tasks into a generalist model. A unified motion model is appealing since it can leverage a wide range of motion data to achieve broad generalization beyond a single task. However, it is also challenging due to the heterogeneous nature of substantially different motion data and tasks. LMM tackles these challenges from three principled aspects: 1) Data: We consolidate datasets with different modalities, formats and tasks into a comprehensive yet unified motion generation dataset, MotionVerse, comprising 10 tasks, 16 datasets, a total of 320k sequences, and 100 million frames. 2) Architecture: We design an articulated attention mechanism ArtAttention that incorporates body part-aware modeling into Diffusion Transformer backbone. 3) Pre-Training: We propose a novel pre-training strategy for LMM, which employs variable frame rates and masking forms, to better exploit knowledge from diverse training data. Extensive experiments demonstrate that our generalist LMM achieves competitive performance across various standard motion generation tasks over state-of-the-art specialist models. Notably, LMM exhibits strong generalization capabilities and emerging properties across many unseen tasks. Additionally, our ablation studies reveal valuable insights about training and scaling up large motion models for future research.