# Text Conditioned Symbolic Drumbeat Generation using Latent Diffusion Models

arXiv: https://arxiv.org/abs/2408.02711 · HTML: https://mlsanity.com/paper/2408.02711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [A Functional Taxonomy of Music Generation Systems](https://mlsanity.com/paper/1812.04186/) (2018)
- [MorpheuS: generating structured music with constrained patterns and tension](https://mlsanity.com/paper/1812.04832/) (2018)
- [Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset](https://mlsanity.com/paper/2004.00188/) (2020)
- [JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAE](https://mlsanity.com/paper/2210.06007/) (2022)
- [Conditional Drums Generation using Compound Word Representations](https://mlsanity.com/paper/2202.04464/) (2022)

## Built on by (most cited first, 2 of 2)

- [Conditional Diffusion as Latent Constraints for Controllable Symbolic Music Generation](https://mlsanity.com/paper/2511.07156/) (2025)
- [Seconds-Aligned PCA-DAC Latent Diffusion for Symbolic-to-Audio Drum Rendering](https://mlsanity.com/paper/2605.13404/) (2026)

## Abstract (arXiv)

This study introduces a text-conditioned approach to generating drumbeats with Latent Diffusion Models (LDMs). It uses informative conditioning text extracted from training data filenames. By pretraining a text and drumbeat encoder through contrastive learning within a multimodal network, aligned following CLIP, we align the modalities of text and music closely. Additionally, we examine an alternative text encoder based on multihot text encodings. Inspired by musics multi-resolution nature, we propose a novel LSTM variant, MultiResolutionLSTM, designed to operate at various resolutions independently. In common with recent LDMs in the image space, it speeds up the generation process by running diffusion in a latent space provided by a pretrained unconditional autoencoder. We demonstrate the originality and variety of the generated drumbeats by measuring distance (both over binary pianorolls and in the latent space) versus the training dataset and among the generated drumbeats. We also assess the generated drumbeats through a listening test focused on questions of quality, aptness for the prompt text, and novelty. We show that the generated drumbeats are novel and apt to the prompt text, and comparable in quality to those created by human musicians.