# Robust Dancer: Long-term 3D Dance Synthesis Using Unpaired Data

arXiv: https://arxiv.org/abs/2303.16856 · HTML: https://mlsanity.com/paper/2303.16856/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer](https://mlsanity.com/paper/2103.10206/) (2021)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [Learning Human Motion Models for Long-term Predictions](https://mlsanity.com/paper/1704.02827/) (2017)
- [Motion Puzzle: Arbitrary Motion Style Transfer by Body Part](https://mlsanity.com/paper/2202.05274/) (2022)
- [Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure](https://mlsanity.com/paper/2111.12159/) (2021)
- [Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio](https://mlsanity.com/paper/2011.12999/) (2020)
- [ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit](https://mlsanity.com/paper/2009.07637/) (2020)
- [Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction](https://mlsanity.com/paper/2205.15608/) (2022)
- [Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network](https://mlsanity.com/paper/1811.00818/) (2018)
- [Music-oriented Dance Video Synthesis with Pose Perceptual Loss](https://mlsanity.com/paper/1912.06606/) (2019)

## Built on by (most cited first, 3 of 3)

- [Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editing](https://mlsanity.com/paper/2410.18977/) (2024)
- [Controllable Group Choreography using Contrastive Diffusion](https://mlsanity.com/paper/2310.18986/) (2023)
- [Scalable Group Choreography via Variational Phase Manifold Learning](https://mlsanity.com/paper/2407.18839/) (2024)

## Abstract (arXiv)

How to automatically synthesize natural-looking dance movements based on a piece of music is an incrementally popular yet challenging task. Most existing data-driven approaches require hard-to-get paired training data and fail to generate long sequences of motion due to error accumulation of autoregressive structure. We present a novel 3D dance synthesis system that only needs unpaired data for training and could generate realistic long-term motions at the same time. For the unpaired data training, we explore the disentanglement of beat and style, and propose a Transformer-based model free of reliance upon paired data. For the synthesis of long-term motions, we devise a new long-history attention strategy. It first queries the long-history embedding through an attention computation and then explicitly fuses this embedding into the generation pipeline via multimodal adaptation gate (MAG). Objective and subjective evaluations show that our results are comparable to strong baseline methods, despite not requiring paired training data, and are robust when inferring long-term music. To our best knowledge, we are the first to achieve unpaired data training - an ability that enables to alleviate data limitations effectively. Our code is released on https://github.com/BFeng14/RobustDancer