# Graph-based Polyphonic Multitrack Music Generation

arXiv: https://arxiv.org/abs/2307.14928 · HTML: https://mlsanity.com/paper/2307.14928/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 27 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs](https://mlsanity.com/paper/1704.02901/) (2017)
- [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017)
- [C-RNN-GAN: Continuous recurrent neural networks with adversarial training](https://mlsanity.com/paper/1611.09904/) (2016)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [A Gentle Introduction to Deep Learning for Graphs](https://mlsanity.com/paper/1912.12693/) (2019)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [Counterpoint by Convolution](https://mlsanity.com/paper/1903.07227/) (2019)
- [MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer](https://mlsanity.com/paper/1809.07600/) (2018)
- [PIANOTREE VAE: Structured Representation Learning for Polyphonic Music](https://mlsanity.com/paper/2008.07118/) (2020)
- [MELONS: generating melody with long-term structure using transformers and structure graph](https://mlsanity.com/paper/2110.05020/) (2021)
- [Song From PI: A Musically Plausible Network for Pop Music Generation](https://mlsanity.com/paper/1611.03477/) (2016)
- [Cadence Detection in Symbolic Classical Music using Graph Neural Networks](https://mlsanity.com/paper/2208.14819/) (2022)
- [JamBot: Music Theory Aware Chord Based Generation of Polyphonic Music with LSTMs](https://mlsanity.com/paper/1711.07682/) (2017)
- [Learning Style-Aware Symbolic Music Representations by Adversarial Autoencoders](https://mlsanity.com/paper/2001.05494/) (2020)
- [On the Complex Network Structure of Musical Pieces: Analysis of Some Use Cases from Different Music Genres](https://mlsanity.com/paper/1709.09708/) (2017)
- [On the Modeling of Musical Solos as Complex Networks](https://mlsanity.com/paper/1610.00468/) (2016)
- [Calliope -- A Polyphonic Music Transformer](https://mlsanity.com/paper/2107.05546/) (2021)

## Built on by (most cited first, 2 of 2)

- [GraphMuse: A Library for Symbolic Music Graph Processing](https://mlsanity.com/paper/2407.12671/) (2024)
- [AnalysisGNN: Unified Music Analysis with Graph Neural Networks](https://mlsanity.com/paper/2509.06654/) (2025)

## Abstract (arXiv)

Graphs can be leveraged to model polyphonic multitrack symbolic music, where notes, chords and entire sections may be linked at different levels of the musical hierarchy by tonal and rhythmic relationships. Nonetheless, there is a lack of works that consider graph representations in the context of deep learning systems for music generation. This paper bridges this gap by introducing a novel graph representation for music and a deep Variational Autoencoder that generates the structure and the content of musical graphs separately, one after the other, with a hierarchical architecture that matches the structural priors of music. By separating the structure and content of musical graphs, it is possible to condition generation by specifying which instruments are played at certain times. This opens the door to a new form of human-computer interaction in the context of music co-creation. After training the model on existing MIDI datasets, the experiments show that the model is able to generate appealing short and long musical sequences and to realistically interpolate between them, producing music that is tonally and rhythmically consistent. Finally, the visualization of the embeddings shows that the model is able to organize its latent space in accordance with known musical concepts.