# Learning Interpretable Representation for Controllable Polyphonic Music Generation

arXiv: https://arxiv.org/abs/2008.07122 · HTML: https://mlsanity.com/paper/2008.07122/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2020.

## Built on by (most cited first, 30 of 40)

- [MuseCoco: Generating Symbolic Music from Text](https://mlsanity.com/paper/2306.00110/) (2023)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [MuPT: A Generative Symbolic Music Pretrained Transformer](https://mlsanity.com/paper/2404.06393/) (2024)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models](https://mlsanity.com/paper/2405.09901/) (2024)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Loop Copilot: Conducting AI Ensembles for Music Generation and Iterative Editing](https://mlsanity.com/paper/2310.12404/) (2023)
- [Self-Supervised VQ-VAE for One-Shot Music Style Transfer](https://mlsanity.com/paper/2102.05749/) (2021)
- [Symbolic music generation conditioned on continuous-valued emotions](https://mlsanity.com/paper/2203.16165/) (2022)
- [Music Composition with Deep Learning: A Review](https://mlsanity.com/paper/2108.12290/) (2021)
- [Structure-Enhanced Pop Music Generation via Harmony-Aware Learning](https://mlsanity.com/paper/2109.06441/) (2021)
- [GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers](https://mlsanity.com/paper/2302.05393/) (2023)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning](https://mlsanity.com/paper/2112.15110/) (2021)
- [Learning long-term music representations via hierarchical contextual constraints](https://mlsanity.com/paper/2202.06180/) (2022)
- [Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation](https://mlsanity.com/paper/2211.05543/) (2022)
- [The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos](https://mlsanity.com/paper/2404.04420/) (2024)
- [Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generation](https://mlsanity.com/paper/2505.03314/) (2025)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models](https://mlsanity.com/paper/2307.13005/) (2023)
- [From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation](https://mlsanity.com/paper/2304.08953/) (2023)
- [Controllable Lyrics-to-Melody Generation](https://mlsanity.com/paper/2306.02613/) (2023)
- [Automatic Time Signature Determination for New Scores Using Lyrics for Latent Rhythmic Structure](https://mlsanity.com/paper/2311.15480/) (2023)
- [MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion](https://mlsanity.com/paper/2312.10307/) (2023)
- [A Survey of Music Generation in the Context of Interaction](https://mlsanity.com/paper/2402.15294/) (2024)
- [Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints](https://mlsanity.com/paper/2407.03824/) (2024)
- [Incorporating Structure and Chord Constraints in Symbolic Transformer-based Melodic Harmonization](https://mlsanity.com/paper/2512.07627/) (2025)
- [Motif-Centric Representation Learning for Symbolic Music](https://mlsanity.com/paper/2309.10597/) (2023)
- [The Interpretation Gap in Text-to-Music Generation Models](https://mlsanity.com/paper/2407.10328/) (2024)

## Abstract (arXiv)

While deep generative models have become the leading methods for algorithmic composition, it remains a challenging problem to control the generation process because the latent variables of most deep-learning models lack good interpretability. Inspired by the content-style disentanglement idea, we design a novel architecture, under the VAE framework, that effectively learns two interpretable latent factors of polyphonic music: chord and texture. The current model focuses on learning 8-beat long piano composition segments. We show that such chord-texture disentanglement provides a controllable generation pathway leading to a wide spectrum of applications, including compositional style transfer, texture variation, and accompaniment arrangement. Both objective and subjective evaluations show that our method achieves a successful disentanglement and high quality controlled music generation.