# Music-oriented Dance Video Synthesis with Pose Perceptual Loss

arXiv: https://arxiv.org/abs/1912.06606 · HTML: https://mlsanity.com/paper/1912.06606/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2019.

## Built on by (most cited first, 5 of 5)

- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [UDE: A Unified Driving Engine for Human Motion Generation](https://mlsanity.com/paper/2211.16016/) (2022)
- [Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks](https://mlsanity.com/paper/2103.06982/) (2021)
- [Music-driven Dance Regeneration with Controllable Key Pose Constraints](https://mlsanity.com/paper/2207.03682/) (2022)
- [Robust Dancer: Long-term 3D Dance Synthesis Using Unpaired Data](https://mlsanity.com/paper/2303.16856/) (2023)

## Abstract (arXiv)

We present a learning-based approach with pose perceptual loss for automatic music video generation. Our method can produce a realistic dance video that conforms to the beats and rhymes of almost any given music. To achieve this, we firstly generate a human skeleton sequence from music and then apply the learned pose-to-appearance mapping to generate the final video. In the stage of generating skeleton sequences, we utilize two discriminators to capture different aspects of the sequence and propose a novel pose perceptual loss to produce natural dances. Besides, we also provide a new cross-modal evaluation to evaluate the dance quality, which is able to estimate the similarity between two modalities of music and dance. Finally, a user study is conducted to demonstrate that dance video synthesized by the presented approach produces surprisingly realistic results. The results are shown in the supplementary video at https://youtu.be/0rMuFMZa_K4