# Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models

arXiv: https://arxiv.org/abs/2405.09901 · HTML: https://mlsanity.com/paper/2405.09901/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 16 May 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 18 of 18)

- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Symbolic Music Generation with Diffusion Models](https://mlsanity.com/paper/2103.16091/) (2021)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [PopMAG: Pop Music Accompaniment Generation](https://mlsanity.com/paper/2008.07703/) (2020)
- [Learning Interpretable Representation for Controllable Polyphonic Music Generation](https://mlsanity.com/paper/2008.07122/) (2020)
- [Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model](https://mlsanity.com/paper/2208.12675/) (2022)
- [Anticipatory Music Transformer](https://mlsanity.com/paper/2306.08620/) (2023)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [Deep Music Analogy Via Latent Representation Disentanglement](https://mlsanity.com/paper/1906.03626/) (2019)
- [Controllable deep melody generation via hierarchical music structure representation](https://mlsanity.com/paper/2109.00663/) (2021)
- [Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythm](https://mlsanity.com/paper/2008.01291/) (2020)
- [Automatic Analysis and Influence of Hierarchical Structure on Melody, Rhythm and Harmony in Popular Music](https://mlsanity.com/paper/2010.07518/) (2020)
- [Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning](https://mlsanity.com/paper/2112.15110/) (2021)
- [Learning long-term music representations via hierarchical contextual constraints](https://mlsanity.com/paper/2202.06180/) (2022)
- [ProtoDiffusion: Classifier-Free Diffusion Guidance with Prototype Learning](https://mlsanity.com/paper/2307.01924/) (2023)

## Built on by (most cited first, 25 of 25)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Autoregressive Diffusion Transformer for Text-to-Speech Synthesis](https://mlsanity.com/paper/2406.05551/) (2024)
- [Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning](https://mlsanity.com/paper/2509.04059/) (2025)
- [Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generation](https://mlsanity.com/paper/2505.03314/) (2025)
- [Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization](https://mlsanity.com/paper/2408.15176/) (2024)
- [BandCondiNet: Parallel Transformers-based Conditional Popular Music Generation with Multi-View Features](https://mlsanity.com/paper/2407.10462/) (2024)
- [Applications and Advances of Artificial Intelligence in Music Generation:A Review](https://mlsanity.com/paper/2409.03715/) (2024)
- [Diffusion-based Symbolic Music Generation with Structured State Space Models](https://mlsanity.com/paper/2507.20128/) (2025)
- [Segment-Factorized Full-Song Generation on Symbolic Piano Music](https://mlsanity.com/paper/2510.05881/) (2025)
- [PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generation](https://mlsanity.com/paper/2512.11348/) (2025)
- [SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton](https://mlsanity.com/paper/2604.25498/) (2026)
- [Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation](https://mlsanity.com/paper/2410.08435/) (2024)
- [Efficient Long-Sequence Diffusion Modeling for Symbolic Music Generation](https://mlsanity.com/paper/2603.00576/) (2026)
- [ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Models](https://mlsanity.com/paper/2603.01984/) (2026)
- [PerceiverS: A Multi-Scale Perceiver with Effective Segmentation for Long-Term Expressive Symbolic Music Generation](https://mlsanity.com/paper/2411.08307/) (2024)
- [Synthesizing Composite Hierarchical Structure from Symbolic Music Corpora](https://mlsanity.com/paper/2502.15849/) (2025)
- [Personalized Federated Training of Diffusion Models with Privacy Guarantees](https://mlsanity.com/paper/2504.00952/) (2025)
- [TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure](https://mlsanity.com/paper/2506.23094/) (2025)
- [Pay (Cross) Attention to the Melody: Curriculum Masking for Single-Encoder Melodic Harmonization](https://mlsanity.com/paper/2601.16150/) (2026)
- [D3PIA: A Discrete Denoising Diffusion Model for Piano Accompaniment Generation From Lead sheet](https://mlsanity.com/paper/2602.03523/) (2026)
- [Structure-Aware Piano Accompaniment via Style Planning and Dataset-Aligned Pattern Retrieval](https://mlsanity.com/paper/2602.15074/) (2026)
- [Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrence](https://mlsanity.com/paper/2602.19816/) (2026)
- [Anchored Cyclic Generation: A Novel Paradigm for Long-Sequence Symbolic Music Generation](https://mlsanity.com/paper/2604.05343/) (2026)
- [Diff-Symbo: Text-Controlled Long-Duration Symbolic Music Generation Using Autoregressive Latent Diffusion Model](https://mlsanity.com/paper/2608.05222/) (2026)
- [Harmonic Representations in Symbolic Music Transformers are Localized and Controllable](https://mlsanity.com/paper/2610.15957/) (2026)

## Abstract (arXiv)

Recent deep music generation studies have put much emphasis on long-term generation with structures. However, we are yet to see high-quality, well-structured whole-song generation. In this paper, we make the first attempt to model a full music piece under the realization of compositional hierarchy. With a focus on symbolic representations of pop songs, we define a hierarchical language, in which each level of hierarchy focuses on the semantics and context dependency at a certain music scope. The high-level languages reveal whole-song form, phrase, and cadence, whereas the low-level languages focus on notes, chords, and their local patterns. A cascaded diffusion model is trained to model the hierarchical language, where each level is conditioned on its upper levels. Experiments and analysis show that our model is capable of generating full-piece music with recognizable global verse-chorus structure and cadences, and the music quality is higher than the baselines. Additionally, we show that the proposed model is controllable in a flexible way. By sampling from the interpretable hierarchical languages or adjusting pre-trained external representations, users can control the music flow via various features such as phrase harmonic structures, rhythmic patterns, and accompaniment texture.