# Temporally Guided Music-to-Body-Movement Generation

arXiv: https://arxiv.org/abs/2009.08015 · HTML: https://mlsanity.com/paper/2009.08015/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 21 of 21)

- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model](https://mlsanity.com/paper/2309.06284/) (2023)
- [TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration](https://mlsanity.com/paper/2304.02419/) (2023)
- [Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure](https://mlsanity.com/paper/2111.12159/) (2021)
- [Quantized GAN for Complex Music Generation from Dance Videos](https://mlsanity.com/paper/2204.00604/) (2022)
- [Fg-T2M++: LLMs-Augmented Fine-Grained Text Driven Human Motion Generation](https://mlsanity.com/paper/2502.05534/) (2025)
- [Motion Inbetweening via Deep Delta-Interpolator](https://mlsanity.com/paper/2201.06701/) (2022)
- [Dance Any Beat: Blending Beats with Visuals in Dance Video Generation](https://mlsanity.com/paper/2405.09266/) (2024)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance](https://mlsanity.com/paper/2410.05791/) (2024)
- [Taming Diffusion Models for Music-driven Conducting Motion Generation](https://mlsanity.com/paper/2306.10065/) (2023)
- [MusicFace: Music-driven Expressive Singing Face Synthesis](https://mlsanity.com/paper/2303.14044/) (2023)
- [Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesis](https://mlsanity.com/paper/2507.06689/) (2025)
- [ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion](https://mlsanity.com/paper/2507.19836/) (2025)
- [MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing](https://mlsanity.com/paper/2406.06375/) (2024)
- [MOST: Motion Diffusion Model for Rare Text via Temporal Clip Banzhaf Interaction](https://mlsanity.com/paper/2507.06590/) (2025)
- [Dance Generation by Sound Symbolic Words](https://mlsanity.com/paper/2306.03646/) (2023)
- [MUSIC: Learning Muscle-Driven Dexterous Hand Control](https://mlsanity.com/paper/2604.23886/) (2026)
- [VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference](https://mlsanity.com/paper/2411.13607/) (2024)

## Abstract (arXiv)

This paper presents a neural network model to generate virtual violinist's 3-D skeleton movements from music audio. Improved from the conventional recurrent neural network models for generating 2-D skeleton data in previous works, the proposed model incorporates an encoder-decoder architecture, as well as the self-attention mechanism to model the complicated dynamics in body movement sequences. To facilitate the optimization of self-attention model, beat tracking is applied to determine effective sizes and boundaries of the training examples. The decoder is accompanied with a refining network and a bowing attack inference mechanism to emphasize the right-hand behavior and bowing attack timing. Both objective and subjective evaluations reveal that the proposed model outperforms the state-of-the-art methods. To the best of our knowledge, this work represents the first attempt to generate 3-D violinists' body movements considering key features in musical body movement.