# Music-driven Dance Regeneration with Controllable Key Pose Constraints

arXiv: https://arxiv.org/abs/2207.03682 · HTML: https://mlsanity.com/paper/2207.03682/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Synthesis of Compositional Animations from Textual Descriptions](https://mlsanity.com/paper/2103.14675/) (2021)
- [DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer](https://mlsanity.com/paper/2103.10206/) (2021)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure](https://mlsanity.com/paper/2111.12159/) (2021)
- [Synthetic Humans for Action Recognition from Unseen Viewpoints](https://mlsanity.com/paper/1912.04070/) (2019)
- [Music-oriented Dance Video Synthesis with Pose Perceptual Loss](https://mlsanity.com/paper/1912.06606/) (2019)
- [Dance Generation with Style Embedding: Learning and Transferring Latent Representations of Dance Styles](https://mlsanity.com/paper/2104.14802/) (2021)

## Built on by (most cited first, 3 of 3)

- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [UniMuMo: Unified Text, Music and Motion Generation](https://mlsanity.com/paper/2410.04534/) (2024)
- [VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference](https://mlsanity.com/paper/2411.13607/) (2024)

## Abstract (arXiv)

In this paper, we propose a novel framework for music-driven dance motion synthesis with controllable key pose constraint. In contrast to methods that generate dance motion sequences only based on music without any other controllable conditions, this work targets on synthesizing high-quality dance motion driven by music as well as customized poses performed by users. Our model involves two single-modal transformer encoders for music and motion representations and a cross-modal transformer decoder for dance motions generation. The cross-modal transformer decoder achieves the capability of synthesizing smooth dance motion sequences, which keeps a consistency with key poses at corresponding positions, by introducing the local neighbor position embedding. Such mechanism makes the decoder more sensitive to key poses and the corresponding positions. Our dance synthesis model achieves satisfactory performance both on quantitative and qualitative evaluations with extensive experiments, which demonstrates the effectiveness of our proposed method.