# Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation

arXiv: https://arxiv.org/abs/2401.08559 · HTML: https://mlsanity.com/paper/2401.08559/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 80 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 30 of 61)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts](https://mlsanity.com/paper/2207.01696/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)

## Built on by (most cited first, 30 of 91)

- [MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model](https://mlsanity.com/paper/2404.19759/) (2024)
- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)
- [Generating Human Interaction Motions in Scenes with Text Control](https://mlsanity.com/paper/2404.10685/) (2024)
- [Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data](https://mlsanity.com/paper/2507.07095/) (2025)
- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [MotionFix: Text-Driven 3D Human Motion Editing](https://mlsanity.com/paper/2408.00712/) (2024)
- [ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model](https://mlsanity.com/paper/2412.14559/) (2024)
- [GENMO: A GENeralist Model for Human MOtion](https://mlsanity.com/paper/2505.01425/) (2025)
- [Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs](https://mlsanity.com/paper/2405.17013/) (2024)
- [MaskControl: Spatio-Temporal Control for Masked Motion Synthesis](https://mlsanity.com/paper/2410.10780/) (2024)
- [SMooDi: Stylized Motion Diffusion Model](https://mlsanity.com/paper/2407.12783/) (2024)
- [HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects](https://mlsanity.com/paper/2407.12371/) (2024)
- [Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfer](https://mlsanity.com/paper/2406.06508/) (2024)
- [Unimotion: Unifying 3D Human Motion Synthesis and Understanding](https://mlsanity.com/paper/2409.15904/) (2024)
- [BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects](https://mlsanity.com/paper/2412.05066/) (2024)
- [EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space](https://mlsanity.com/paper/2412.14706/) (2024)
- [MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities](https://mlsanity.com/paper/2504.02478/) (2025)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation](https://mlsanity.com/paper/2412.06781/) (2024)
- [KinMo: Kinematic-aware Human Motion Understanding and Generation](https://mlsanity.com/paper/2411.15472/) (2024)
- [Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing](https://mlsanity.com/paper/2410.18977/) (2024)
- [KMM: Key Frame Mask Mamba for Extended Motion Generation](https://mlsanity.com/paper/2411.06481/) (2024)
- [Simulating the Real World: A Unified Survey of Multimodal Generative Models](https://mlsanity.com/paper/2503.04641/) (2025)
- [Dynamic Motion Blending for Versatile Motion Editing](https://mlsanity.com/paper/2503.20724/) (2025)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning](https://mlsanity.com/paper/2504.04956/) (2025)
- [SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control](https://mlsanity.com/paper/2412.15664/) (2024)
- [Chronologically Accurate Retrieval for Temporal Grounding of Motion-Language Models](https://mlsanity.com/paper/2407.15408/) (2024)
- [Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control](https://mlsanity.com/paper/2503.14517/) (2025)
- [SceneMI: Motion In-betweening for Modeling Human-Scene Interactions](https://mlsanity.com/paper/2503.16289/) (2025)

## Abstract (arXiv)

Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts. Our code and models are publicly available at https://mathis.petrovich.fr/stmc.