mlsanityData to 7 Oct 2026

Scaling Large Motion Models with Million-Level Human Motions

arXiv 2410.03311 ↗First posted 4 Oct 2024cs.CV cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Inspired by the recent success of LLMs, the field of human motion understanding has increasingly shifted toward developing large motion models. Despite some progress, current efforts remain far from achieving truly generalist models, primarily due to the lack of massive high-quality data. To address this gap, we present MotionLib, the first million-level dataset for motion generation, which is at least 15× larger than existing counterparts and enriched with hierarchical text descriptions. Using MotionLib, we train a large motion model named projname, demonstrating robust performance across a wide range of human activities, including unseen ones. Through systematic investigation, for the first time, we highlight the importance of scaling both data and model size for advancing motion generation, along with key insights to achieve this goal. To better integrate the motion modality, we propose Motionbook, an innovative motion encoding approach including (1) a compact yet lossless feature to represent motions; (2) a novel 2D lookup-free motion tokenizer that preserves fine-grained motion details while expanding codebook capacity, significantly enhancing the representational power of motion tokens. We believe this work lays the groundwork for developing more versatile and powerful motion generation models in the future. For further details, visit https://beingbeyond.github.io/Being-M0/.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 59 of its 72 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

59 of 72 references.

References · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2014Going Deeper with Convolutionscited by 4,231
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  7. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  10. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  11. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  12. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  13. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  14. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  15. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  16. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  17. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  18. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  19. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  20. 2016The KIT Motion-Language Datasetcited by 334
  21. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
  22. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  23. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  24. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  25. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  26. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 200
  27. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  28. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  29. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  30. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  31. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  32. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  33. 2017Learning Human Motion Models for Long-term Predictionscited by 67
  34. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  35. 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
  36. 2022Implicit Neural Representations for Variable Length Human Motion Generationcited by 61
  37. 2022Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formatscited by 50
  38. 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 46

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  2. 2025Being-M0.5: A Real-Time Controllable Vision-Language-Motion Modelcited by 9
  3. 2025UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectoriescited by 6
  4. 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 4
  5. 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videoscited by 4
  6. 2025A Scalable Whole-body Motion Transfer via Implicit Kinodynamic Motion Retargetingcited by 2
  7. 2026RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generationcited by 2
  8. 2025Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Modelscited by 1
  9. 2026CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wildcited by 1
  10. 2026Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Controlcited by 1
  11. 2026Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUscited by 1
  12. 2026AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modelingcited by 1
8 more
  1. 2026FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervisioncited by 1
  2. 2025Less is More: Improving Motion Diffusion Models with Sparse Keyframescited by 0
  3. 2025Cognitive State Inference from VR Motion via Motion Foundation Modelcited by 0
  4. 2025FunPhase: A Periodic Functional Autoencoder for Motion Generation via Phase Manifoldscited by 0
  5. 2025Robust Motion Generation using Part-level Reliable Data from Videoscited by 0
  6. 2026DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcingcited by 0
  7. 2026XmoPipe: A Pipeline for Large-Scale In-the-Wild Human Motion Dataset Constructioncited by 0
  8. 2026World Motion Models: Flexible Sequence Modeling of SE(3) Trajectoriescited by 0