# Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

arXiv: https://arxiv.org/abs/2405.17013 · HTML: https://mlsanity.com/paper/2405.17013/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 48 of its 56 references that are in the map.
- First posted on arXiv 27 May 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 48)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)

## Built on by (most cited first, 30 of 47)

- [SMooDi: Stylized Motion Diffusion Model](https://mlsanity.com/paper/2407.12783/) (2024)
- [CoMA: Compositional Human Motion Generation with Multi-modal Agents](https://mlsanity.com/paper/2412.07320/) (2024)
- [LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration](https://mlsanity.com/paper/2504.00010/) (2025)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens](https://mlsanity.com/paper/2602.12370/) (2026)
- [MoLA: Motion Generation and Editing with Latent Diffusion Enhanced by Adversarial Training](https://mlsanity.com/paper/2406.01867/) (2024)
- [Motion-R1: Enhancing Motion Generation with Decomposed Chain-of-Thought and RL Binding](https://mlsanity.com/paper/2506.10353/) (2025)
- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)
- [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025)
- [GenM3: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generation](https://mlsanity.com/paper/2503.14919/) (2025)
- [FSBench: A Figure Skating Benchmark for Advancing Artistic Sports Understanding](https://mlsanity.com/paper/2504.19514/) (2025)
- [IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation](https://mlsanity.com/paper/2512.10730/) (2025)
- [SafeMo: Linguistically Grounded Unlearning for Trustworthy Text-to-Motion Generation](https://mlsanity.com/paper/2601.00590/) (2026)
- [CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos](https://mlsanity.com/paper/2601.10632/) (2026)
- [A Unified Framework for Motion Reasoning and Generation in Human Interaction](https://mlsanity.com/paper/2410.05628/) (2024)
- [DanceChat: Large Language Model-Guided Music-to-Dance Generation](https://mlsanity.com/paper/2506.10574/) (2025)
- [VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models](https://mlsanity.com/paper/2508.12081/) (2025)
- [HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA](https://mlsanity.com/paper/2511.01463/) (2025)
- [SentiAvatar: Towards Expressive and Interactive Digital Humans](https://mlsanity.com/paper/2604.02908/) (2026)
- [MotionHiFlow: Text-to-motion via hierarchical flow matching](https://mlsanity.com/paper/2604.23264/) (2026)
- [Dense Motion Captioning](https://mlsanity.com/paper/2511.05369/) (2025)
- [UniMo: Unified Motion Generation and Understanding with Chain of Thought](https://mlsanity.com/paper/2601.12126/) (2026)
- [PRISM: Streaming Human Motion Generation with Per-Joint Latent Decomposition](https://mlsanity.com/paper/2603.08590/) (2026)
- [Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning](https://mlsanity.com/paper/2503.07323/) (2025)
- [PlanMoGPT: Flow-Enhanced Progressive Planning for Text to Motion Synthesis](https://mlsanity.com/paper/2506.17912/) (2025)
- [U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation](https://mlsanity.com/paper/2602.23739/) (2026)
- [CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning](https://mlsanity.com/paper/2605.28056/) (2026)
- [Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward](https://mlsanity.com/paper/2506.03191/) (2025)
- [OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions](https://mlsanity.com/paper/2512.19159/) (2025)
- [UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars](https://mlsanity.com/paper/2605.14731/) (2026)

## Abstract (arXiv)

While previous approaches to 3D human motion generation have achieved notable success, they often rely on extensive training and are limited to specific tasks. To address these challenges, we introduce Motion-Agent, an efficient conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text. This is accomplished by encoding and quantizing motions into discrete tokens that align with the language model's vocabulary. With only 1--3% of the model's parameters fine-tuned using adapters, MotionLLM delivers performance on par with diffusion models and other transformer-based methods trained from scratch. By integrating MotionLLM with GPT-4 without additional training, Motion-Agent is able to generate highly complex motion sequences through multi-turn conversations, a capability that previous models have struggled to achieve. Motion-Agent supports a wide range of motion-language tasks, offering versatile capabilities for generating and customizing human motion through interactive conversational exchanges. Project page: https://knoxzhao.github.io/Motion-Agent