Scaling Large Motion Models with Million-Level Human Motions
Not on the map. Why?
Abstract (arXiv)
Inspired by the recent success of LLMs, the field of human motion understanding has increasingly shifted toward developing large motion models. Despite some progress, current efforts remain far from achieving truly generalist models, primarily due to the lack of massive high-quality data. To address this gap, we present MotionLib, the first million-level dataset for motion generation, which is at least 15× larger than existing counterparts and enriched with hierarchical text descriptions. Using MotionLib, we train a large motion model named projname, demonstrating robust performance across a wide range of human activities, including unseen ones. Through systematic investigation, for the first time, we highlight the importance of scaling both data and model size for advancing motion generation, along with key insights to achieve this goal. To better integrate the motion modality, we propose Motionbook, an innovative motion encoding approach including (1) a compact yet lossless feature to represent motions; (2) a novel 2D lookup-free motion tokenizer that preserves fine-grained motion details while expanding codebook capacity, significantly enhancing the representational power of motion tokens. We believe this work lays the groundwork for developing more versatile and powerful motion generation models in the future. For further details, visit https://beingbeyond.github.io/Being-M0/.
Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 59 of its 72 references that are in the map. First posted on arXiv 4 Oct 2024.
Builds on
59 of 72 references.
References · 59
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Segment Anythingcited by 8,158
- 2023Visual Instruction Tuningcited by 7,792
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2017Neural Discrete Representation Learningcited by 4,650
- 2014Going Deeper with Convolutionscited by 4,231
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2017The Kinetics Human Action Video Datasetcited by 1,894
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
- 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
- 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
- 2022Autoregressive Image Generation using Residual Quantizationcited by 706
- 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
- 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
- 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
- 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
- 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
- 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
- 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
- 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
- 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
- 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
- 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
- 2016The KIT Motion-Language Datasetcited by 334
- 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
- 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
- 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
- 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
- 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 200
- 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
- 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
- 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
- 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
- 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
- 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
- 2017Learning Human Motion Models for Long-term Predictionscited by 67
- 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
- 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
- 2022Implicit Neural Representations for Variable Length Human Motion Generationcited by 61
- 2022Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formatscited by 50
- 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 46
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2025: 10; 2026: 10.
20 papers.
Sort:
By area:
- 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
- 2025Being-M0.5: A Real-Time Controllable Vision-Language-Motion Modelcited by 9
- 2025UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectoriescited by 6
- 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 4
- 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videoscited by 4
- 2025A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargetingcited by 2
- 2026RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generationcited by 2
- 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Modelscited by 1
- 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
- 2026Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Controlcited by 1
- 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
- 2026AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modelingcited by 1
8 more
- 2026FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervisioncited by 1
- 2025Less is More: Improving Motion Diffusion Models with Sparse Keyframescited by 0
- 2025Cognitive State Inference from VR Motion via Motion Foundation Modelcited by 0
- 2025FunPhase: A Periodic Functional Autoencoder for Motion Generation via Phase Manifoldscited by 0
- 2025Robust Motion Generation using Part-level Reliable Data from Videoscited by 0
- 2026DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcingcited by 0
- 2026XmoPipe: A Pipeline for Large-Scale In-the-Wild Human Motion Dataset Constructioncited by 0
- 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectoriescited by 0
- 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories1 Oct 2026
- 2026FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision2 Aug 2026
- 2026XmoPipe: A Pipeline for Large-Scale In-the-Wild Human Motion Dataset Construction17 Jun 2026
- 2026AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling28 May 2026
- 2026DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing27 May 2026
- 2026RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generation25 May 2026
- 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs23 Apr 2026
- 2026Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Control25 Feb 2026
- 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild19 Feb 2026
- 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos15 Jan 2026
- 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models24 Dec 2025
- 2025Robust Motion Generation using Part-level Reliable Data from Videos14 Dec 2025
8 more
- 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation11 Dec 2025
- 2025UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectories10 Dec 2025
- 2025FunPhase: A Periodic Functional Autoencoder for Motion Generation via Phase Manifolds10 Dec 2025
- 2025Cognitive State Inference from VR Motion via Motion Foundation Model29 Sep 2025
- 2025A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargeting18 Sep 2025
- 2025Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model11 Aug 2025
- 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos21 Jul 2025
- 2025Less is More: Improving Motion Diffusion Models with Sparse Keyframes18 Mar 2025
- 2025Being-M0.5: A Real-Time Controllable Vision-Language-Motion Modelcited by 9
- 2025UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectoriescited by 6
- 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 4
- 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videoscited by 4
- 2026RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generationcited by 2
- 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Modelscited by 1
- 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
- 2026Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Controlcited by 1
- 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
- 2026AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modelingcited by 1
- 2026FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervisioncited by 1
- 2025Cognitive State Inference from VR Motion via Motion Foundation Modelcited by 0
12 most cited of 17 All 17 (JSON)
- 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories1 Oct 2026
- 2026FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision2 Aug 2026
- 2026XmoPipe: A Pipeline for Large-Scale In-the-Wild Human Motion Dataset Construction17 Jun 2026
- 2026AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling28 May 2026
- 2026DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcing27 May 2026
- 2026RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generation25 May 2026
- 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs23 Apr 2026
- 2026Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Control25 Feb 2026
- 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild19 Feb 2026
- 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos15 Jan 2026
- 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models24 Dec 2025
- 2025Robust Motion Generation using Part-level Reliable Data from Videos14 Dec 2025
12 newest of 17 All 17 (JSON)