# ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

arXiv: https://arxiv.org/abs/2607.08741 · HTML: https://mlsanity.com/paper/2607.08741/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 73 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Builds on (most cited first, 30 of 45)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [Mean Flows for One-step Generative Modeling](https://mlsanity.com/paper/2505.13447/) (2025)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [Action2Motion: Conditioned Generation of 3D Human Motions](https://mlsanity.com/paper/2007.15240/) (2020)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models](https://mlsanity.com/paper/2410.11081/) (2024)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Model](https://mlsanity.com/paper/2404.19759/) (2024)
- [BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion](https://mlsanity.com/paper/2508.08241/) (2025)
- [Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion](https://mlsanity.com/paper/2304.01893/) (2023)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control](https://mlsanity.com/paper/2410.03441/) (2024)
- [Optimizing Diffusion Noise Can Serve As Universal Motion Priors](https://mlsanity.com/paper/2312.11994/) (2023)
- [TLControl: Trajectory and Language Control for Human Motion Synthesis](https://mlsanity.com/paper/2311.17135/) (2023)
- [Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation](https://mlsanity.com/paper/2401.08559/) (2024)
- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)

## Built on by (most cited first, 7 of 7)

- [Marionette: Predicting World States, Rendering Geometry, Painting Appearance](https://mlsanity.com/paper/2608.14530/) (2026)
- [Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions](https://mlsanity.com/paper/2609.16683/) (2026)
- [Dancing Stick Figures: An Introductory Dataset for Training Video Generation Models](https://mlsanity.com/paper/2608.29123/) (2026)
- [PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control](https://mlsanity.com/paper/2609.24840/) (2026)
- [FloodDiffusion 2: Efficient and Path Controllable Streaming Motion Generation](https://mlsanity.com/paper/2609.33167/) (2026)
- [Streaming Multi-Track Timeline Control for 3D Human Motion Generation](https://mlsanity.com/paper/2610.03873/) (2026)
- [Sonic-AMG: Streaming Human Motion Generation from Spatial Audio](https://mlsanity.com/paper/2610.13300/) (2026)

## Abstract (arXiv)

Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.