# DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation

arXiv: https://arxiv.org/abs/2308.02915 · HTML: https://mlsanity.com/paper/2308.02915/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 5 Aug 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis](https://mlsanity.com/paper/2204.09934/) (2022)
- [Human Motion Prediction via Spatio-Temporal Inpainting](https://mlsanity.com/paper/1812.05478/) (2018)
- [Deep representation learning for human motion prediction and classification](https://mlsanity.com/paper/1702.07486/) (2017)
- [Structured Prediction Helps 3D Human Motion Modelling](https://mlsanity.com/paper/1910.09070/) (2019)
- [Learning to Generate Diverse Dance Motions with Transformer](https://mlsanity.com/paper/2008.08171/) (2020)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [Audio to Body Dynamics](https://mlsanity.com/paper/1712.09382/) (2017)
- [Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio](https://mlsanity.com/paper/2011.12999/) (2020)

## Built on by (most cited first, 23 of 23)

- [Large Motion Model for Unified Multi-Modal Motion Generation](https://mlsanity.com/paper/2404.01284/) (2024)
- [VideoVLA: Video Generators Can Be Generalizable Robot Manipulators](https://mlsanity.com/paper/2512.06963/) (2025)
- [DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer](https://mlsanity.com/paper/2402.05712/) (2024)
- [SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization](https://mlsanity.com/paper/2412.05095/) (2024)
- [Dance Any Beat: Blending Beats with Visuals in Dance Video Generation](https://mlsanity.com/paper/2405.09266/) (2024)
- [SceneMI: Motion In-betweening for Modeling Human-Scene Interactions](https://mlsanity.com/paper/2503.16289/) (2025)
- [Motion Synthesis with Sparse and Flexible Keyjoint Control](https://mlsanity.com/paper/2503.15557/) (2025)
- [Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representation](https://mlsanity.com/paper/2503.17340/) (2025)
- [Text2Grasp: Grasp synthesis by text prompts of object grasping parts](https://mlsanity.com/paper/2404.15189/) (2024)
- [Lagrangian Motion Fields for Long-term Motion Generation](https://mlsanity.com/paper/2409.01522/) (2024)
- [MusicInfuser: Making Video Diffusion Listen and Dance](https://mlsanity.com/paper/2503.14505/) (2025)
- [Scalable Group Choreography via Variational Phase Manifold Learning](https://mlsanity.com/paper/2407.18839/) (2024)
- [ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion](https://mlsanity.com/paper/2507.19836/) (2025)
- [GCDance: Genre-Controlled Music-Driven 3D Full Body Dance Generation](https://mlsanity.com/paper/2502.18309/) (2025)
- [ReactDance: Hierarchical Representation for High-Fidelity and Coherent Long-Form Reactive Dance Generation](https://mlsanity.com/paper/2505.05589/) (2025)
- [MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation](https://mlsanity.com/paper/2506.02661/) (2025)
- [ST-GDance: Long-Term and Collision-Free Group Choreography from Music](https://mlsanity.com/paper/2507.21518/) (2025)
- [How Animals Dance (When You're Not Looking)](https://mlsanity.com/paper/2505.23738/) (2025)
- [Reframing Music-Driven 2D Dance Pose Generation as Multi-Channel Image Generation](https://mlsanity.com/paper/2512.11720/) (2025)
- [Motion Capture is Not the Target Domain: Scaling Synthetic Data for Learning Motion Representations](https://mlsanity.com/paper/2602.11064/) (2026)
- [ST-GDance++: A Scalable Spatial-Temporal Diffusion for Long-Duration Group Choreography](https://mlsanity.com/paper/2603.22316/) (2026)
- [TeMuDance: Contrastive Alignment-Based Textual Control for Music-Driven Dance Generation](https://mlsanity.com/paper/2604.17005/) (2026)
- [FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams](https://mlsanity.com/paper/2608.15818/) (2026)

## Abstract (arXiv)

When hearing music, it is natural for people to dance to its rhythm. Automatic dance generation, however, is a challenging task due to the physical constraints of human motion and rhythmic alignment with target music. Conventional autoregressive methods introduce compounding errors during sampling and struggle to capture the long-term structure of dance sequences. To address these limitations, we present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. To bridge the gap between music and motion for conditional generation, DiffDance employs a pretrained audio representation learning model to extract music embeddings and further align its embedding space to motion via contrastive loss. During training our cascaded diffusion model, we also incorporate multiple geometric losses to constrain the model outputs to be physically plausible and add a dynamic loss weight that adaptively changes over diffusion timesteps to facilitate sample diversity. Through comprehensive experiments performed on the benchmark dataset AIST++, we demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music. These results are comparable to those achieved by state-of-the-art autoregressive methods.