# Is Disentanglement enough? On Latent Representations for Controllable Music Generation

arXiv: https://arxiv.org/abs/2108.01450 · HTML: https://mlsanity.com/paper/2108.01450/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 1 Aug 2021.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [Learning Deep Disentangled Embeddings with the F-Statistic Loss](https://mlsanity.com/paper/1802.05312/) (2018)
- [DeepBach: a Steerable Model for Bach Chorales Generation](https://mlsanity.com/paper/1612.01010/) (2016)
- [Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer](https://mlsanity.com/paper/1807.07543/) (2018)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling](https://mlsanity.com/paper/2007.15474/) (2020)
- [Counterpoint by Convolution](https://mlsanity.com/paper/1903.07227/) (2019)
- [Music transcription modelling and composition using deep learning](https://mlsanity.com/paper/1604.08723/) (2016)
- [MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer](https://mlsanity.com/paper/1809.07600/) (2018)
- [Deep Music Analogy Via Latent Representation Disentanglement](https://mlsanity.com/paper/1906.03626/) (2019)
- [Music Generation by Deep Learning - Challenges and Directions](https://mlsanity.com/paper/1712.04371/) (2017)
- [Learning to Traverse Latent Spaces for Musical Score Inpainting](https://mlsanity.com/paper/1907.01164/) (2019)
- [GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures](https://mlsanity.com/paper/1707.04588/) (2017)
- [Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1906.08152/) (2019)
- [Representing Closed Transformation Paths in Encoded Network Latent Space](https://mlsanity.com/paper/1912.02644/) (2019)
- [Piano Genie](https://mlsanity.com/paper/1810.05246/) (2018)
- [Musical Composition Style Transfer via Disentangled Timbre Representations](https://mlsanity.com/paper/1905.13567/) (2019)
- [NONOTO: A Model-agnostic Web Interface for Interactive Music Composition by Inpainting](https://mlsanity.com/paper/1907.10380/) (2019)
- [dMelodies: A Music Dataset for Disentanglement Learning](https://mlsanity.com/paper/2007.15067/) (2020)
- [Algorithmic Composition of Melodies with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1606.07251/) (2016)

## Built on by (most cited first, 7 of 7)

- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [MusIAC: An extensible generative framework for Music Infilling Applications with multi-level Control](https://mlsanity.com/paper/2202.05528/) (2022)
- [Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes](https://mlsanity.com/paper/2110.05587/) (2021)
- [Evaluating Disentangled Representations for Controllable Music Generation](https://mlsanity.com/paper/2602.10058/) (2026)
- [Latte: Cross-framework Python Package for Evaluation of Latent-Based Generative Models](https://mlsanity.com/paper/2112.10638/) (2021)
- [AnchorSteer: Self-Discovered Concept Injection for Structure-Preserving Music Editing](https://mlsanity.com/paper/2605.31053/) (2026)

## Abstract (arXiv)

Improving controllability or the ability to manipulate one or more attributes of the generated data has become a topic of interest in the context of deep generative models of music. Recent attempts in this direction have relied on learning disentangled representations from data such that the underlying factors of variation are well separated. In this paper, we focus on the relationship between disentanglement and controllability by conducting a systematic study using different supervised disentanglement learning algorithms based on the Variational Auto-Encoder (VAE) architecture. Our experiments show that a high degree of disentanglement can be achieved by using different forms of supervision to train a strong discriminative encoder. However, in the absence of a strong generative decoder, disentanglement does not necessarily imply controllability. The structure of the latent space with respect to the VAE-decoder plays an important role in boosting the ability of a generative model to manipulate different attributes. To this end, we also propose methods and metrics to help evaluate the quality of a latent space with respect to the afforded degree of controllability.