# Music to Dance as Language Translation using Sequence Models

arXiv: https://arxiv.org/abs/2403.15569 · HTML: https://mlsanity.com/paper/2403.15569/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2024.

## Built on by (most cited first, 3 of 3)

- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba](https://mlsanity.com/paper/2603.27314/) (2026)
- [BiTDiff: Fine-Grained 3D Conducting Motion Generation via BiMamba-Transformer Diffusion](https://mlsanity.com/paper/2604.04395/) (2026)

## Abstract (arXiv)

Synthesising appropriate choreographies from music remains an open problem. We introduce MDLT, a novel approach that frames the choreography generation problem as a translation task. Our method leverages an existing data set to learn to translate sequences of audio into corresponding dance poses. We present two variants of MDLT: one utilising the Transformer architecture and the other employing the Mamba architecture. We train our method on AIST++ and PhantomDance data sets to teach a robotic arm to dance, but our method can be applied to a full humanoid robot. Evaluation metrics, including Average Joint Error and Fr'echet Inception Distance, consistently demonstrate that, when given a piece of music, MDLT excels at producing realistic and high-quality choreography. The code can be found at github.com/meowatthemoon/MDLT.