# PIANOTREE VAE: Structured Representation Learning for Polyphonic Music

arXiv: https://arxiv.org/abs/2008.07118 · HTML: https://mlsanity.com/paper/2008.07118/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2020.

## Built on by (most cited first, 25 of 25)

- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Multitrack Music Transformer](https://mlsanity.com/paper/2207.06983/) (2022)
- [A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling](https://mlsanity.com/paper/2212.00973/) (2022)
- [Music Composition with Deep Learning: A Review](https://mlsanity.com/paper/2108.12290/) (2021)
- [Structure-Enhanced Pop Music Generation via Harmony-Aware Learning](https://mlsanity.com/paper/2109.06441/) (2021)
- [JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAE](https://mlsanity.com/paper/2210.06007/) (2022)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling](https://mlsanity.com/paper/2310.16334/) (2023)
- [Exploring XAI for the Arts: Explaining Latent Space in Generative Music](https://mlsanity.com/paper/2308.05496/) (2023)
- [Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning](https://mlsanity.com/paper/2112.15110/) (2021)
- [Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generation](https://mlsanity.com/paper/2408.01180/) (2024)
- [Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation](https://mlsanity.com/paper/2211.05543/) (2022)
- [Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI](https://mlsanity.com/paper/2311.08336/) (2023)
- [α-TCVAE: On the relationship between Disentanglement and Diversity](https://mlsanity.com/paper/2411.00588/) (2024)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [BEAT: Tokenizing and Generating Symbolic Music by Uniform Temporal Steps](https://mlsanity.com/paper/2604.19532/) (2026)
- [An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation](https://mlsanity.com/paper/2301.13383/) (2023)
- [Graph-based Polyphonic Multitrack Music Generation](https://mlsanity.com/paper/2307.14928/) (2023)
- [Multi-view MidiVAE: Fusing Track- and Bar-view Representations for Long Multi-track Symbolic Music Generation](https://mlsanity.com/paper/2401.07532/) (2024)
- [PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generation](https://mlsanity.com/paper/2512.11348/) (2025)
- [Flat Latent Manifolds for Human-machine Co-creation of Music](https://mlsanity.com/paper/2202.12243/) (2022)
- [Motif-Centric Representation Learning for Symbolic Music](https://mlsanity.com/paper/2309.10597/) (2023)
- [From Generality to Mastery: Composer-Style Symbolic Music Generation via Large-Scale Pre-training](https://mlsanity.com/paper/2506.17497/) (2025)
- [MuseTok: Symbolic Music Tokenization for Generation and Semantic Understanding](https://mlsanity.com/paper/2510.16273/) (2025)

## Abstract (arXiv)

The dominant approach for music representation learning involves the deep unsupervised model family variational autoencoder (VAE). However, most, if not all, viable attempts on this problem have largely been limited to monophonic music. Normally composed of richer modality and more complex musical structures, the polyphonic counterpart has yet to be addressed in the context of music representation learning. In this work, we propose the PianoTree VAE, a novel tree-structure extension upon VAE aiming to fit the polyphonic music learning. The experiments prove the validity of the PianoTree VAE via (i)-semantically meaningful latent code for polyphonic segments; (ii)-more satisfiable reconstruction aside of decent geometry learned in the latent space; (iii)-this model's benefits to the variety of the downstream music generation.