# Exploring how a Generative AI interprets music

arXiv: https://arxiv.org/abs/2308.00015 · HTML: https://mlsanity.com/paper/2308.00015/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Discovering physical concepts with neural networks](https://mlsanity.com/paper/1807.10300/) (2018)
- [Detecting Symmetries with Neural Networks](https://mlsanity.com/paper/2003.13679/) (2020)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [Discovering State Variables Hidden in Experimental Data](https://mlsanity.com/paper/2112.10755/) (2021)
- [A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics](https://mlsanity.com/paper/1811.11440/) (2018)
- [Classical Music Prediction and Composition by means of Variational Autoencoders](https://mlsanity.com/paper/1906.09972/) (2019)

## Built on by (most cited first, 4 of 4)

- [Opening Musical Creativity? Embedded Ideologies in Generative-AI Music Systems](https://mlsanity.com/paper/2508.08805/) (2025)
- [Learning symmetries in datasets](https://mlsanity.com/paper/2504.05174/) (2025)
- [Artificial Intelligence and Symmetries: Learning, Encoding, and Discovering Structure in Physical Data](https://mlsanity.com/paper/2602.02351/) (2026)
- [Latent Spectroscopy: Posterior Collapse as a Feature](https://mlsanity.com/paper/2605.22691/) (2026)

## Abstract (arXiv)

We use Google's MusicVAE, a Variational Auto-Encoder with a 512-dimensional latent space to represent a few bars of music, and organize the latent dimensions according to their relevance in describing music. We find that, on average, most latent neurons remain silent when fed real music tracks: we call these "noise" neurons. The remaining few dozens of latent neurons that do fire are called "music neurons". We ask which neurons carry the musical information and what kind of musical information they encode, namely something that can be identified as pitch, rhythm or melody. We find that most of the information about pitch and rhythm is encoded in the first few music neurons: the neural network has thus constructed a couple of variables that non-linearly encode many human-defined variables used to describe pitch and rhythm. The concept of melody only seems to show up in independent neurons for longer sequences of music.