# Do Music Generation Models Encode Music Theory?

arXiv: https://arxiv.org/abs/2410.00872 · HTML: https://mlsanity.com/paper/2410.00872/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [A Foundation Model for Music Informatics](https://mlsanity.com/paper/2311.03318/) (2023)
- [Does Vision-and-Language Pretraining Improve Lexical Grounding?](https://mlsanity.com/paper/2109.10246/) (2021)
- [Melody transcription via generative pre-training](https://mlsanity.com/paper/2212.01884/) (2022)
- [What Do Language Models Hear? Probing for Auditory Representations in Language Models](https://mlsanity.com/paper/2402.16998/) (2024)
- [Emergence of Abstract State Representations in Embodied Sequence Modeling](https://mlsanity.com/paper/2311.02171/) (2023)
- [JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs](https://mlsanity.com/paper/1711.07682/) (2017)

## Built on by (most cited first, 13 of 13)

- [Discovering and Steering Interpretable Concepts in Large Generative Music Models](https://mlsanity.com/paper/2505.18186/) (2025)
- [Fourier Head: Helping Large Language Models Learn Complex Probability Distributions](https://mlsanity.com/paper/2410.22269/) (2024)
- [Steering Autoregressive Music Generation with Recursive Feature Machines](https://mlsanity.com/paper/2510.19127/) (2025)
- [TADA! Tuning Audio Diffusion Models through Activation Steering](https://mlsanity.com/paper/2602.11910/) (2026)
- [Evaluating Disentangled Representations for Controllable Music Generation](https://mlsanity.com/paper/2602.10058/) (2026)
- [MusGU+: Toward a Musician-Centered Evaluation Framework and Discovery Tool for Generative Music AI](https://mlsanity.com/paper/2608.30940/) (2026)
- [Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping](https://mlsanity.com/paper/2608.03050/) (2026)
- [MI-MIDI: Mechanistic Interpretability of Text-to-MIDI Generation Models via Probing, Lenses and Steering](https://mlsanity.com/paper/2608.06638/) (2026)
- [What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models](https://mlsanity.com/paper/2608.14819/) (2026)
- [Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes](https://mlsanity.com/paper/2609.04516/) (2026)
- [Ghost in the Encoder: Decodable Artist Identity Representations in Lyrics-to-Song Generation](https://mlsanity.com/paper/2609.39552/) (2026)
- [From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment](https://mlsanity.com/paper/2610.01864/) (2026)
- [Harmonic Representations in Symbolic Music Transformers are Localized and Controllable](https://mlsanity.com/paper/2610.15957/) (2026)

## Abstract (arXiv)

Music foundation models possess impressive music generation capabilities. When people compose music, they may infuse their understanding of music into their work, by using notes and intervals to craft melodies, chords to build progressions, and tempo to create a rhythmic feel. To what extent is this true of music generation models? More specifically, are fundamental Western music theory concepts observable within the "inner workings" of these models? Recent work proposed leveraging latent audio representations from music generation models towards music information retrieval tasks (e.g. genre classification, emotion recognition), which suggests that high-level musical characteristics are encoded within these models. However, probing individual music theory concepts (e.g. tempo, pitch class, chord quality) remains under-explored. Thus, we introduce SynTheory, a synthetic MIDI and audio music theory dataset, consisting of tempos, time signatures, notes, intervals, scales, chords, and chord progressions concepts. We then propose a framework to probe for these music theory concepts in music foundation models (Jukebox and MusicGen) and assess how strongly they encode these concepts within their internal representations. Our findings suggest that music theory concepts are discernible within foundation models and that the degree to which they are detectable varies by model size and layer.