# ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit

arXiv: https://arxiv.org/abs/2009.07637 · HTML: https://mlsanity.com/paper/2009.07637/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2020.

## Built on by (most cited first, 30 of 31)

- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation](https://mlsanity.com/paper/2212.03741/) (2022)
- [Seamless Human Motion Composition with Blended Positional Encodings](https://mlsanity.com/paper/2402.15509/) (2024)
- [TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration](https://mlsanity.com/paper/2304.02419/) (2023)
- [Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure](https://mlsanity.com/paper/2111.12159/) (2021)
- [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023)
- [Transflower: probabilistic autoregressive dance generation with multimodal attention](https://mlsanity.com/paper/2106.13871/) (2021)
- [Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns](https://mlsanity.com/paper/2410.20389/) (2024)
- [Dance with You: The Diversity Controllable Dancer Generation via Diffusion Models](https://mlsanity.com/paper/2308.13551/) (2023)
- [Controllable Group Choreography using Contrastive Diffusion](https://mlsanity.com/paper/2310.18986/) (2023)
- [Dance Any Beat: Blending Beats with Visuals in Dance Video Generation](https://mlsanity.com/paper/2405.09266/) (2024)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrations](https://mlsanity.com/paper/2312.11442/) (2023)
- [DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance](https://mlsanity.com/paper/2403.13667/) (2024)
- [Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses](https://mlsanity.com/paper/2312.12042/) (2023)
- [MusicFace: Music-driven Expressive Singing Face Synthesis](https://mlsanity.com/paper/2303.14044/) (2023)
- [MusicInfuser: Making Video Diffusion Listen and Dance](https://mlsanity.com/paper/2503.14505/) (2025)
- [LM2D: Lyrics- and Music-Driven Dance Synthesis](https://mlsanity.com/paper/2403.09407/) (2024)
- [Scalable Group Choreography via Variational Phase Manifold Learning](https://mlsanity.com/paper/2407.18839/) (2024)
- [Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis](https://mlsanity.com/paper/2507.06689/) (2025)
- [ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion](https://mlsanity.com/paper/2507.19836/) (2025)
- [Dual Learning Music Composition and Dance Choreography](https://mlsanity.com/paper/2201.11999/) (2022)
- [Robust Dancer: Long-term 3D Dance Synthesis Using Unpaired Data](https://mlsanity.com/paper/2303.16856/) (2023)
- [DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis](https://mlsanity.com/paper/2409.14925/) (2024)
- [Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation](https://mlsanity.com/paper/2512.18804/) (2025)
- [TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba](https://mlsanity.com/paper/2603.27314/) (2026)
- [Dance Generation by Sound Symbolic Words](https://mlsanity.com/paper/2306.03646/) (2023)
- [Diverse Dance Synthesis via Keyframes with Transformer Controllers](https://mlsanity.com/paper/2207.05906/) (2022)

## Abstract (arXiv)

Dance and music are two highly correlated artistic forms. Synthesizing dance motions has attracted much attention recently. Most previous works conduct music-to-dance synthesis via directly music to human skeleton keypoints mapping. Meanwhile, human choreographers design dance motions from music in a two-stage manner: they firstly devise multiple choreographic dance units (CAUs), each with a series of dance motions, and then arrange the CAU sequence according to the rhythm, melody and emotion of the music. Inspired by these, we systematically study such two-stage choreography approach and construct a dataset to incorporate such choreography knowledge. Based on the constructed dataset, we design a two-stage music-to-dance synthesis framework ChoreoNet to imitate human choreography procedure. Our framework firstly devises a CAU prediction model to learn the mapping relationship between music and CAU sequences. Afterwards, we devise a spatial-temporal inpainting model to convert the CAU sequence into continuous dance motions. Experimental results demonstrate that the proposed ChoreoNet outperforms baseline methods (0.622 in terms of CAU BLEU score and 1.59 in terms of user study score).