# Motifs, Phrases, and Beyond: The Modelling of Structure in Symbolic Music Generation

arXiv: https://arxiv.org/abs/2403.07995 · HTML: https://mlsanity.com/paper/2403.07995/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 78 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets](https://mlsanity.com/paper/1811.03761/) (2018)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [This Time with Feeling: Learning Expressive Musical Performance](https://mlsanity.com/paper/1808.03715/) (2018)
- [LakhNES: Improving multi-instrumental music generation with cross-domain pre-training](https://mlsanity.com/paper/1907.04868/) (2019)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer](https://mlsanity.com/paper/2111.04093/) (2021)
- [Symphony Generation with Permutation Invariant Language Model](https://mlsanity.com/paper/2205.05448/) (2022)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [MorpheuS: generating structured music with constrained patterns and tension](https://mlsanity.com/paper/1812.04832/) (2018)
- [AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer](https://mlsanity.com/paper/2108.11213/) (2021)
- [Automatic Analysis and Influence of Hierarchical Structure on Melody, Rhythm and Harmony in Popular Music](https://mlsanity.com/paper/2010.07518/) (2020)
- [TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method](https://mlsanity.com/paper/2109.09617/) (2021)
- [MELONS: generating melody with long-term structure using transformers and structure graph](https://mlsanity.com/paper/2110.05020/) (2021)
- [Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach](https://mlsanity.com/paper/2209.08212/) (2022)
- [A Hierarchical Recurrent Neural Network for Symbolic Melody Generation](https://mlsanity.com/paper/1712.05274/) (2017)
- [Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure](https://mlsanity.com/paper/2102.09794/) (2021)
- [Structure-Enhanced Pop Music Generation via Harmony-Aware Learning](https://mlsanity.com/paper/2109.06441/) (2021)
- [MelodyGLM: Multi-task Pre-training for Symbolic Melody Generation](https://mlsanity.com/paper/2309.10738/) (2023)
- [Conditional Drums Generation using Compound Word Representations](https://mlsanity.com/paper/2202.04464/) (2022)
- [MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks](https://mlsanity.com/paper/2208.14345/) (2022)
- [WuYun: Exploring hierarchical skeleton-guided melody generation using knowledge-enhanced deep learning](https://mlsanity.com/paper/2301.04488/) (2023)

## Built on by (most cited first, 7 of 7)

- [ImprovNet -- Generating Controllable Musical Improvisations with Iterative Corruption Refinement](https://mlsanity.com/paper/2502.04522/) (2025)
- [Watermarking for AI Content Detection: A Review on Text, Visual, and Audio Modalities](https://mlsanity.com/paper/2504.03765/) (2025)
- [F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation](https://mlsanity.com/paper/2502.10491/) (2025)
- [Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation](https://mlsanity.com/paper/2603.00576/) (2026)
- [Synthesizing Composite Hierarchical Structure from Symbolic Music Corpora](https://mlsanity.com/paper/2502.15849/) (2025)
- [Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation](https://mlsanity.com/paper/2504.05364/) (2025)
- [The Ghost in the Keys: A Disklavier Demo for Human-AI Musical Co-Creativity](https://mlsanity.com/paper/2511.01663/) (2025)

## Abstract (arXiv)

Modelling musical structure is vital yet challenging for artificial intelligence systems that generate symbolic music compositions. This literature review dissects the evolution of techniques for incorporating coherent structure, from symbolic approaches to foundational and transformative deep learning methods that harness the power of computation and data across a wide variety of training paradigms. In the later stages, we review an emerging technique which we refer to as "sub-task decomposition" that involves decomposing music generation into separate high-level structural planning and content creation stages. Such systems incorporate some form of musical knowledge or neuro-symbolic methods by extracting melodic skeletons or structural templates to guide the generation. Progress is evident in capturing motifs and repetitions across all three eras reviewed, yet modelling the nuanced development of themes across extended compositions in the style of human composers remains difficult. We outline several key future directions to realize the synergistic benefits of combining approaches from all eras examined.