PhraseVAE and PhraseLDM: Latent Diffusion for Full-Song Multitrack Symbolic Music Generation
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
This technical report presents a new paradigm for full-song symbolic music generation. Existing symbolic models operate on note-attribute tokens and suffer from extremely long sequences, limited context length, and weak support for long-range structure. We address these issues by introducing PhraseVAE and PhraseLDM, the first latent diffusion framework designed for full-song multitrack symbolic music. PhraseVAE compresses an arbitrary variable-length polyphonic note sequence into a single compact 64-dimensional phrase-level latent representation with high reconstruction fidelity, allowing a well-structured latent space and efficient generative modeling. Built on this latent space, PhraseLDM generates an entire multi-track song in a single pass without any autoregressive components. The system eliminates bar-wise sequential modeling, supports up to 128 bars of music (8 minutes at 64 bpm), and produces complete songs with coherent local texture, idiomatic instrument patterns, and clear global structure. With only 45M parameters, our framework generates a full song within seconds while maintaining competitive musical quality and generation diversity. Together, these results show that phrase-level latent diffusion provides an effective and scalable solution to long-sequence modeling in symbolic music generation. We hope this work encourages future symbolic music research to move beyond note-attribute tokens and to consider phrase-level units as a more effective and musically meaningful modeling target.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 37 of its 47 references that are in the map. First posted on arXiv 12 Dec 2025.
Builds on
37 of 47 references.
Closest ancestors · 4
Same direction, newest first
- 2025Diffusion-based Symbolic Music Generation with Structured State Space Models27 Jul
- 2024Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models16 May
- 2023Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls19 Jul
- 2021Symbolic Music Generation with Diffusion Models30 Mar
Other references · 33
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022Scalable Diffusion Models with Transformerscited by 5,931
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2018Music Transformercited by 207
- 2024Stable Audio Opencited by 192
21 more
- 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
- 2023On Memorization in Diffusion Modelscited by 122
- 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 109
- 2020POP909: A Pop-song Dataset for Music Arrangement Generationcited by 83
- 2020PopMAG: Pop Music Accompaniment Generationcited by 68
- 2023MuseCoco: Generating Symbolic Music from Textcited by 50
- 2023Anticipatory Music Transformercited by 35
- 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 32
- 2019Deep Music Analogy Via Latent Representation Disentanglementcited by 31
- 2025NotaGen: Advancing Musicality in Symbolic Music Generation with Large Language Model Training Paradigmscited by 29
- 2020PIANOTREE VAE: Structured Representation Learning for Polyphonic Musiccited by 25
- 2023Byte Pair Encoding for Symbolic Musiccited by 19
- 2024MelodyT5: A Unified Score-to-Score Transformer for Symbolic Music Processingcited by 12
- 2024CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Modelscited by 12
- 2024Frechet Music Distance: A Metric For Generative Symbolic Music Evaluationcited by 12
- 2023Structured Multi-Track Accompaniment Arrangement via Style Prior Modellingcited by 9
- 2025MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Compositioncited by 7
- 2025Moonbeam: A MIDI Foundation Model Using Both Absolute and Relative Music Attributescited by 4
- 2024Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenizationcited by 3
- 2025Versatile Symbolic Music-for-Music Modeling via Function Alignmentcited by 1
- 2025Segment-Factorized Full-Song Generation on Symbolic Piano Musiccited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Diff-Symbo: Text-Controlled Long-Duration Symbolic Music Generation Using Autoregressive Latent Diffusion Model
- 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusion
- 2024Multi-Track MusicLDM: Towards Versatile Music Generation with Latent Diffusion Model
- 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement
- 2024Why Perturbing Symbolic Music is Necessary: Fitting the Distribution of Never-used Notes through a Joint Probabilistic Diffusion Model
- 2024Long-form music generation with latent diffusion
- 2025Conditional Diffusion as Latent Constraints for Controllable Symbolic Music Generation
- 2025ProGress: Structured Music Generation via Graph Diffusion and Hierarchical Music Analysis