mlsanityData to 7 Oct 2026

MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators

arXiv 2306.10900 ↗First posted 19 Jun 2023cs.CV cs.AIcited by 145

Not on the map. Why?

Abstract (arXiv)

Generating realistic human motion from given action descriptions has experienced significant advancements because of the emerging requirement of digital humans. While recent works have achieved impressive results in generating motion directly from textual action descriptions, they often support only a single modality of the control signal, which limits their application in the real digital human industry. This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals, e.g., text and single-frame poses, for generating consecutive human motions by treating multimodal signals as special input tokens in large language models (LLMs). Specifically, we first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction to ask the LLMs to generate the motion answer. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters. To the best of our knowledge, MotionGPT is the first method to generate human motion by multimodal control signals, which we hope can shed light on this new direction. Visit our webpage at https://qiqiapink.github.io/MotionGPT/.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 19 Jun 2023.

Builds on

42 of 56 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
30 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2022Human Motion Diffusion Modelcited by 1,062
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  15. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  16. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  17. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  18. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  19. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  20. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  21. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  22. 2017On human motion prediction using recurrent neural networkscited by 336
  23. 2016The KIT Motion-Language Datasetcited by 334
  24. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  25. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  26. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  27. 2023Vision-Language Models as Success Detectorscited by 138
  28. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  29. 2021TrajeVAE: Controllable Human Motion Generation from Trajectoriescited by 14
  30. 2020Music2Dance: DanceNet for Music-driven Dance Generationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 54; 2025: 47; 2026: 29.

145 papers.

Sort:

By area:

  1. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  2. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  3. 2023Large Language Models as General Pattern Machinescited by 196
  4. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  5. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  6. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  7. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  8. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  9. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  10. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 67
  11. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  12. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
38 more
  1. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  2. 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63
  3. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  4. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  5. 2024LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioningcited by 58
  6. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 58
  7. 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 52
  8. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  9. 2024InterMask: 3D Human Interaction Generation via Collaborative Masked Modelingcited by 47
  10. 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 46
  11. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  12. 2024MoGenTS: Motion Generation based on Spatial-Temporal Joint Modelingcited by 40
  13. 2025Motion Anything: Any to Motion Generationcited by 40
  14. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  15. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  16. 2024M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generationcited by 37
  17. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  18. 2025SALAD: Skeleton-aware Latent Diffusion for Text-driven Motion Generation and Editingcited by 37
  19. 2024LLMs are Good Action Recognizerscited by 36
  20. 2025MotionGPT3: Human Motion as a Second Modalitycited by 36
  21. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  22. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 32
  23. 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 32
  24. 2025MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularitiescited by 29
  25. 2024EgoLM: Multi-Modal Language Model of Egocentric Motionscited by 28
  26. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  27. 2025Human Motion Video Generation: A Surveycited by 25
  28. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  29. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 22
  30. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 21
  31. 2025DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modelingcited by 21
  32. 2024Scaling Large Motion Models with Million-Level Human Motionscited by 20
  33. 2024Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generatorcited by 20
  34. 2023What is the Best Automated Metric for Text to Motion Generation?cited by 19
  35. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19
  36. 2025CAD-Editor: A Locate-then-Infill Framework with Automated Training Data Synthesis for Text-Based CAD Editingcited by 19
  37. 2024UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editingcited by 18
  38. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18

50 most cited of 145 All 145 (JSON)