# DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Control

arXiv: https://arxiv.org/abs/2410.05260 · HTML: https://mlsanity.com/paper/2410.05260/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 87 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 65)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset](https://mlsanity.com/paper/2307.00818/) (2023)

## Built on by (most cited first, 30 of 84)

- [MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Space](https://mlsanity.com/paper/2503.15451/) (2025)
- [Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation](https://mlsanity.com/paper/2512.04678/) (2025)
- [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [Kimodo: Scaling Controllable Human Motion Generation](https://mlsanity.com/paper/2603.15546/) (2026)
- [HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation](https://mlsanity.com/paper/2512.23464/) (2025)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control](https://mlsanity.com/paper/2504.12540/) (2025)
- [The Quest for Generalizable Motion Generation: Data, Model, and Evaluation](https://mlsanity.com/paper/2510.26794/) (2025)
- [Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis](https://mlsanity.com/paper/2508.02106/) (2025)
- [PRIMAL: Physically Reactive and Interactive Motor Model for Avatar Learning](https://mlsanity.com/paper/2503.17544/) (2025)
- [TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Control](https://mlsanity.com/paper/2602.07439/) (2026)
- [SCENIC: Scene-aware Semantic Navigation with Instruction-guided Control](https://mlsanity.com/paper/2412.15664/) (2024)
- [EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer](https://mlsanity.com/paper/2512.18814/) (2025)
- [Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy](https://mlsanity.com/paper/2503.18349/) (2025)
- [FrankenMotion: Part-level Human Motion Generation and Composition](https://mlsanity.com/paper/2601.10909/) (2026)
- [Causal Motion Diffusion Models for Autoregressive Motion Generation](https://mlsanity.com/paper/2602.22594/) (2026)
- [VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisions](https://mlsanity.com/paper/2506.23236/) (2025)
- [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://mlsanity.com/paper/2507.05419/) (2025)
- [Future Optical Flow Prediction Improves Robot Control & Video Generation](https://mlsanity.com/paper/2601.10781/) (2026)
- [Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Models](https://mlsanity.com/paper/2512.19692/) (2025)
- [DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotion](https://mlsanity.com/paper/2505.18780/) (2025)
- [UniMoGen: Universal Motion Generation](https://mlsanity.com/paper/2505.21837/) (2025)
- [ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planning](https://mlsanity.com/paper/2603.13500/) (2026)
- [Heracles: Bridging Precise Tracking and Generative Synthesis for General Humanoid Control](https://mlsanity.com/paper/2603.27756/) (2026)
- [Controllable Human-centric Keyframe Interpolation with Generative Prior](https://mlsanity.com/paper/2506.03119/) (2025)
- [Moving by Looking: Towards Vision-Driven Avatar Motion Generation](https://mlsanity.com/paper/2509.19259/) (2025)
- [OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation](https://mlsanity.com/paper/2510.19789/) (2025)
- [AHA! Animating Human Avatars in Diverse Scenes with Gaussian Splatting](https://mlsanity.com/paper/2511.09827/) (2025)
- [HINT: Hierarchical Interaction Modeling for Autoregressive Multi-Human Motion Generation](https://mlsanity.com/paper/2601.20383/) (2026)

## Abstract (arXiv)

Text-conditioned human motion generation, which allows for user interaction through natural language, has become increasingly popular. Existing methods typically generate short, isolated motions based on a single input sentence. However, human motions are continuous and can extend over long periods, carrying rich semantics. Creating long, complex motions that precisely respond to streams of text descriptions, particularly in an online and real-time setting, remains a significant challenge. Furthermore, incorporating spatial constraints into text-conditioned motion generation presents additional challenges, as it requires aligning the motion semantics specified by text descriptions with geometric information, such as goal locations and 3D scene geometry. To address these limitations, we propose DartControl, in short DART, a Diffusion-based Autoregressive motion primitive model for Real-time Text-driven motion control. Our model effectively learns a compact motion primitive space jointly conditioned on motion history and text inputs using latent diffusion models. By autoregressively generating motion primitives based on the preceding history and current text input, DART enables real-time, sequential motion generation driven by natural language descriptions. Additionally, the learned motion primitive space allows for precise spatial motion control, which we formulate either as a latent noise optimization problem or as a Markov decision process addressed through reinforcement learning. We present effective algorithms for both approaches, demonstrating our model's versatility and superior performance in various motion synthesis tasks. Experiments show our method outperforms existing baselines in motion realism, efficiency, and controllability. Video results are available on the project page: https://zkf1997.github.io/DART/.