# Language Models are Drummers: Drum Composition with Natural Language Pre-Training

arXiv: https://arxiv.org/abs/2301.01162 · HTML: https://mlsanity.com/paper/2301.01162/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [Learning to Groove with Inverse Sequence Transformations](https://mlsanity.com/paper/1905.06118/) (2019)
- [Symbolic Music Genre Transfer with CycleGAN](https://mlsanity.com/paper/1809.07575/) (2018)
- [PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music](https://mlsanity.com/paper/2010.08091/) (2020)
- [High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interaction](https://mlsanity.com/paper/1908.00948/) (2019)
- [Dance Dance Convolution](https://mlsanity.com/paper/1703.06891/) (2017)
- [Artificial Neural Networks Jamming on the Beat](https://mlsanity.com/paper/2007.06284/) (2020)

## Built on by (most cited first, 3 of 3)

- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Not that Groove: Zero-Shot Symbolic Music Editing](https://mlsanity.com/paper/2505.08203/) (2025)
- [JAMMIN-GPT: Text-based Improvisation using LLMs in Ableton Live](https://mlsanity.com/paper/2312.03479/) (2023)

## Abstract (arXiv)

Automatic music generation with artificial intelligence typically requires a large amount of data which is hard to obtain for many less common genres and musical instruments. To tackle this issue, we present ongoing work and preliminary findings on the possibility for deep models to transfer knowledge from language to music, by finetuning large language models pre-trained on a massive text corpus on only hundreds of MIDI files of drum performances. We show that by doing so, one of the largest, state-of-the-art models (GPT3) is capable of generating reasonable drum grooves, while models that are not pre-trained (Transformer) shows no such ability beyond naive repetition. Evaluating generated music is a challenging task, more so is evaluating drum grooves with little precedence in literature. Hence, we propose a tailored structural evaluation method and analyze drum grooves produced by GPT3 compared to those played by human professionals, exposing the strengths and weaknesses of such generation by language-to-music transfer. Our findings suggest that language-to-music transfer learning with large language models is viable and promising.