# Generalized Audio-Driven Synthesis of Precise Drummer Motion

arXiv: https://arxiv.org/abs/2608.19055 · HTML: https://mlsanity.com/paper/2608.19055/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 19 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [GENMO: A GENeralist Model for Human MOtion](https://mlsanity.com/paper/2505.01425/) (2025)
- [Learning to Groove with Inverse Sequence Transformations](https://mlsanity.com/paper/1905.06118/) (2019)
- [A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation](https://mlsanity.com/paper/2203.09893/) (2022)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization](https://mlsanity.com/paper/2506.15625/) (2025)
- [Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset](https://mlsanity.com/paper/2004.00188/) (2020)
- [Omnizart: A General Toolbox for Automatic Music Transcription](https://mlsanity.com/paper/2106.00497/) (2021)
- [F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance](https://mlsanity.com/paper/2410.05791/) (2024)
- [Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing](https://mlsanity.com/paper/2409.16629/) (2024)
- [ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition](https://mlsanity.com/paper/2505.04203/) (2025)
- [PAMD: Plausibility-Aware Motion Diffusion Model for Long Dance Generation](https://mlsanity.com/paper/2505.20056/) (2025)
- [Robot Drummer: Learning Rhythmic Skills for Humanoid Drumming](https://mlsanity.com/paper/2507.11498/) (2025)

## Abstract (arXiv)

Music-driven character animation enables and enhances transformative applications in entertainment and interactive education. However, synthesizing realistic drumming motion from audio remains challenging due to the inherent tension between high-acceleration dynamics and the need for extreme spatial-temporal precision. Existing approaches, often reliant on motion matching or MIDI input, struggle with generalizing to diverse real-world audio. Moreover, the field lacks standardized evaluation metrics capable of distinguishing precise drumming from noisy motion. In this paper, we introduce a generative diffusion framework featuring a dual-objective loss function that decouples skeletal integrity from drumstick precision, thus enabling centimeter-level stick precision without sacrificing natural body dynamics. Additionally, leveraging our own dataset and data augmentation strategy, the model generalizes to non-curated, in-the-wild audio. To rigorously evaluate performance, we propose two novel metrics: an impact-to-target distance to quantify spatial precision and an audio-motion correlation score to assess temporal alignment. Our quantitative analysis and user studies demonstrate that our system generates high-quality motion that is often indistinguishable from ground-truth performances.