# CoMA: Compositional Human Motion Generation with Multi-modal Agents

arXiv: https://arxiv.org/abs/2412.07320 · HTML: https://mlsanity.com/paper/2412.07320/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 44 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [MMM: Generative Masked Motion Model](https://mlsanity.com/paper/2312.03596/) (2023)
- [FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing](https://mlsanity.com/paper/2312.15004/) (2023)
- [Optimizing Diffusion Noise Can Serve As Universal Motion Priors](https://mlsanity.com/paper/2312.11994/) (2023)
- [Attribute2Image: Conditional Image Generation from Visual Attributes](https://mlsanity.com/paper/1512.00570/) (2015)

## Built on by (most cited first, 19 of 19)

- [Generative Trajectory Stitching through Diffusion Composition](https://mlsanity.com/paper/2503.05153/) (2025)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [Multi-Modal Manipulation via Multi-Modal Policy Consensus](https://mlsanity.com/paper/2509.23468/) (2025)
- [PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMs](https://mlsanity.com/paper/2508.17188/) (2025)
- [Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agents](https://mlsanity.com/paper/2510.04637/) (2025)
- [VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning](https://mlsanity.com/paper/2511.17731/) (2025)
- [Let EEG Models Learn EEG](https://mlsanity.com/paper/2605.21280/) (2026)
- [Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models](https://mlsanity.com/paper/2511.19917/) (2025)
- [Dense Motion Captioning](https://mlsanity.com/paper/2511.05369/) (2025)
- [LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents](https://mlsanity.com/paper/2512.17445/) (2025)
- [The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination](https://mlsanity.com/paper/2601.08237/) (2026)
- [HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles](https://mlsanity.com/paper/2602.21333/) (2026)
- [Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning](https://mlsanity.com/paper/2503.07323/) (2025)
- [FineMoLA: Towards Fine-Grained Motion-Language Alignment from Clip-Level Supervision](https://mlsanity.com/paper/2608.01392/) (2026)
- [ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer](https://mlsanity.com/paper/2503.21847/) (2025)
- [MoRL: Reinforced Reasoning for Unified Motion Understanding and Generation](https://mlsanity.com/paper/2602.14534/) (2026)
- [Test-Time Registers as Global Priors for Tokenized Image Generation](https://mlsanity.com/paper/2607.16824/) (2026)
- [COSMI: COmpositional Synthesis of Multi-object Interactions](https://mlsanity.com/paper/2610.03252/) (2026)

## Abstract (arXiv)

3D human motion generation has seen substantial advancement in recent years. While state-of-the-art approaches have improved performance significantly, they still struggle with complex and detailed motions unseen in training data, largely due to the scarcity of motion datasets and the prohibitive cost of generating new training examples. To address these challenges, we introduce CoMA, an agent-based solution for complex human motion generation, editing, and comprehension. CoMA leverages multiple collaborative agents powered by large language and vision models, alongside a mask transformer-based motion generator featuring body part-specific encoders and codebooks for fine-grained control. Our framework enables generation of both short and long motion sequences with detailed instructions, text-guided motion editing, and self-correction for improved quality. Evaluations on the HumanML3D dataset demonstrate competitive performance against state-of-the-art methods. Additionally, we create a set of context-rich, compositional, and long text prompts, where user studies show our method significantly outperforms existing approaches.