# Towards Open Domain Text-Driven Synthesis of Multi-Person Motions

arXiv: https://arxiv.org/abs/2405.18483 · HTML: https://mlsanity.com/paper/2405.18483/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 90 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)

## Built on by (most cited first, 29 of 29)

- [DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control](https://mlsanity.com/paper/2410.05260/) (2024)
- [InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling](https://mlsanity.com/paper/2410.10010/) (2024)
- [A Survey on Human Interaction Motion Generation](https://mlsanity.com/paper/2503.12763/) (2025)
- [DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling](https://mlsanity.com/paper/2506.18680/) (2025)
- [SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesis](https://mlsanity.com/paper/2412.20104/) (2024)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [PINO: Person-Interaction Noise Optimization for Long-Duration and Customizable Motion Generation of Arbitrary-Sized Groups](https://mlsanity.com/paper/2507.19292/) (2025)
- [Multi-Person Interaction Generation from Two-Person Motion Priors](https://mlsanity.com/paper/2505.17860/) (2025)
- [Pulp Motion: Framing-aware multimodal camera and human motion generation](https://mlsanity.com/paper/2510.05097/) (2025)
- [3D Human Interaction Generation: A Survey](https://mlsanity.com/paper/2503.13120/) (2025)
- [Ponimator: Unfolding Interactive Pose for Versatile Human-human Interaction Animation](https://mlsanity.com/paper/2510.14976/) (2025)
- [Ig3D: Integrating 3D Face Representations in Facial Expression Inference](https://mlsanity.com/paper/2408.16907/) (2024)
- [MotionVerse: A Unified Multimodal Framework for Motion Comprehension, Generation and Editing](https://mlsanity.com/paper/2509.23635/) (2025)
- [Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene](https://mlsanity.com/paper/2507.19232/) (2025)
- [Next-Scale Autoregressive Models for Text-to-Motion Generation](https://mlsanity.com/paper/2604.03799/) (2026)
- [Leader and Follower: Interactive Motion Generation under Trajectory Constraints](https://mlsanity.com/paper/2502.11563/) (2025)
- [Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation](https://mlsanity.com/paper/2603.00144/) (2026)
- [Learning to Generate Human-Human-Object Interactions from Textual Descriptions](https://mlsanity.com/paper/2511.20446/) (2025)
- [AMG: Avatar Motion Guided Video Generation](https://mlsanity.com/paper/2409.01502/) (2024)
- [ReMoGen: Real-time Human Interaction-to-Reaction Generation via Modular Learning from Diverse Data](https://mlsanity.com/paper/2604.01082/) (2026)
- [Talking Together: Synthesizing Co-Located 3D Conversations from Audio](https://mlsanity.com/paper/2603.08674/) (2026)
- [ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos](https://mlsanity.com/paper/2603.17267/) (2026)
- [Unified Number-Free Text-to-Motion Generation Via Flow Matching](https://mlsanity.com/paper/2603.27040/) (2026)
- [Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers](https://mlsanity.com/paper/2604.01128/) (2026)
- [PhysiGen: Integrating Collision-Aware Physical Constraints for High-Fidelity Human-Human Interaction Generation](https://mlsanity.com/paper/2605.00517/) (2026)
- [GNOCHI: Generative Neural mOdel for Close Human-Human Interactions](https://mlsanity.com/paper/2607.10408/) (2026)
- [A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models](https://mlsanity.com/paper/2608.15984/) (2026)
- [COSMI: COmpositional Synthesis of Multi-object Interactions](https://mlsanity.com/paper/2610.03252/) (2026)
- [Scalable Multi-Human Cooperative Manipulation via Commander-Executor Coordination](https://mlsanity.com/paper/2610.14911/) (2026)

## Abstract (arXiv)

This work aims to generate natural and diverse group motions of multiple humans from textual descriptions. While single-person text-to-motion generation is extensively studied, it remains challenging to synthesize motions for more than one or two subjects from in-the-wild prompts, mainly due to the lack of available datasets. In this work, we curate human pose and motion datasets by estimating pose information from large-scale image and video datasets. Our models use a transformer-based diffusion framework that accommodates multiple datasets with any number of subjects or frames. Experiments explore both generation of multi-person static poses and generation of multi-person motion sequences. To our knowledge, our method is the first to generate multi-subject motion sequences with high diversity and fidelity from a large variety of textual prompts.