# Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades

arXiv: https://arxiv.org/abs/2603.08028 · HTML: https://mlsanity.com/paper/2603.08028/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 61 of its 78 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 61)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [Executing your Commands via Motion Diffusion in Latent Space](https://mlsanity.com/paper/2212.04048/) (2022)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023)
- [Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance](https://mlsanity.com/paper/2403.14781/) (2024)
- [MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance](https://mlsanity.com/paper/2406.19680/) (2024)
- [DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion](https://mlsanity.com/paper/2304.06025/) (2023)
- [HunyuanVideo 1.5 Technical Report](https://mlsanity.com/paper/2511.18870/) (2025)
- [ControlNeXt: Powerful and Efficient Control for Image and Video Generation](https://mlsanity.com/paper/2408.06070/) (2024)
- [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://mlsanity.com/paper/2411.17440/) (2024)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion](https://mlsanity.com/paper/2311.12052/) (2023)
- [ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference](https://mlsanity.com/paper/2407.12442/) (2024)

## Abstract (arXiv)

Generating videos of complex human motions such as flips, cartwheels, and martial arts remains challenging for current video diffusion models. Text-only conditioning is temporally ambiguous for fine-grained motion control, while explicit pose-based controls, though effective, require users to provide complete skeleton sequences that are costly to produce for long and dynamic actions. We propose a two-stage cascaded framework that addresses both limitations. First, an autoregressive text-to-skeleton model generates 2D pose sequences from natural language descriptions by predicting each joint conditioned on previously generated poses. This design captures long-range temporal dependencies and inter-joint coordination required for complex motions. Second, a pose-conditioned video diffusion model synthesizes videos from a reference image and the generated skeleton sequence. It employs DINO-ALF (Adaptive Layer Fusion), a multi-level reference encoder that preserves appearance and clothing details under large pose changes and self-occlusions. To address the lack of publicly available datasets for complex human motion video generation, we introduce a Blender-based synthetic dataset containing 2,000 videos with diverse characters performing acrobatic and stunt-like motions. The dataset provides full control over appearance, motion, and environment. It fills an important gap because existing benchmarks significantly under-represent acrobatic motions while web-collected datasets raise copyright and privacy concerns. Experiments on our synthetic dataset and the Motion-X Fitness benchmark show that our text-to-skeleton model outperforms prior methods on FID, R-precision, and motion diversity. Our pose-to-video model also achieves the best results among all compared methods on VBench metrics for temporal consistency, motion smoothness, and subject preservation.