# Practical and Reproducible Symbolic Music Generation by Large Language Models with Structural Embeddings

arXiv: https://arxiv.org/abs/2407.19900 · HTML: https://mlsanity.com/paper/2407.19900/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [POP909: A Pop-song Dataset for Music Arrangement Generation](https://mlsanity.com/paper/2008.07142/) (2020)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [The Jazz Transformer on the Front Line: Exploring the Shortcomings of AI-composed Music through Quantitative Measures](https://mlsanity.com/paper/2008.01307/) (2020)
- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer](https://mlsanity.com/paper/2111.04093/) (2021)
- [GiantMIDI-Piano: A large-scale MIDI dataset for classical piano music](https://mlsanity.com/paper/2010.07061/) (2020)
- [A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling](https://mlsanity.com/paper/2212.00973/) (2022)
- [Structure-Enhanced Pop Music Generation via Harmony-Aware Learning](https://mlsanity.com/paper/2109.06441/) (2021)

## Built on by (most cited first, 1 of 1)

- [LK Jam: System Architecture and Implementation of a Real-Time Human-AI Interactive Music Generation System using Role-Aware GRU](https://mlsanity.com/paper/2606.21018/) (2026)

## Abstract (arXiv)

Music generation introduces challenging complexities to large language models. Symbolic structures of music often include vertical harmonization as well as horizontal counterpoint, urging various adaptations and enhancements for large-scale Transformers. However, existing works share three major drawbacks: 1) their tokenization requires domain-specific annotations, such as bars and beats, that are typically missing in raw MIDI data; 2) the pure impact of enhancing token embedding methods is hardly examined without domain-specific annotations; and 3) existing works to overcome the aforementioned drawbacks, such as MuseNet, lack reproducibility. To tackle such limitations, we develop a MIDI-based music generation framework inspired by MuseNet, empirically studying two structural embeddings that do not rely on domain-specific annotations. We provide various metrics and insights that can guide suitable encoding to deploy. We also verify that multiple embedding configurations can selectively boost certain musical aspects. By providing open-source implementations via HuggingFace, our findings shed light on leveraging large language models toward practical and reproducible music generation.