# SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition

arXiv: https://arxiv.org/abs/2402.17645 · HTML: https://mlsanity.com/paper/2402.17645/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://mlsanity.com/paper/2105.02446/) (2021)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis](https://mlsanity.com/paper/2201.07429/) (2022)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Conditional LSTM-GAN for Melody Generation from Lyrics](https://mlsanity.com/paper/1908.05551/) (2019)
- [All-In-One Metrical And Functional Structure Analysis With Neighborhood Attentions on Demixed Audio](https://mlsanity.com/paper/2307.16425/) (2023)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling](https://mlsanity.com/paper/2107.01875/) (2021)
- [DopeLearning: A Computational Approach to Rap Lyrics Generation](https://mlsanity.com/paper/1505.04771/) (2015)
- [NHSS: A Speech and Singing Parallel Database](https://mlsanity.com/paper/2012.00337/) (2020)
- [A Hierarchical Recurrent Neural Network for Symbolic Melody Generation](https://mlsanity.com/paper/1712.05274/) (2017)
- [JVS-MuSiC: Japanese multispeaker singing-voice corpus](https://mlsanity.com/paper/2001.07044/) (2020)
- [ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships](https://mlsanity.com/paper/2207.05688/) (2022)
- [What is missing in deep music generation? A study of repetition and structure in popular music](https://mlsanity.com/paper/2209.00182/) (2022)
- [Neural Melody Composition from Lyrics](https://mlsanity.com/paper/1809.04318/) (2018)
- [Youling: an AI-Assisted Lyrics Creation System](https://mlsanity.com/paper/2201.06724/) (2022)

## Built on by (most cited first, 9 of 9)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls](https://mlsanity.com/paper/2412.09887/) (2024)
- [SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training](https://mlsanity.com/paper/2412.18107/) (2024)
- [MusiChat: Vibe Composing for Music Creation](https://mlsanity.com/paper/2607.24873/) (2026)
- [Harmonic Reasoning in Large Language Models](https://mlsanity.com/paper/2409.05521/) (2024)
- [TuneGenie: Reasoning-based LLM agents for preferential music generation](https://mlsanity.com/paper/2506.12083/) (2025)
- [SLEEPING-DISCO 9M: A large-scale pre-training dataset for generative music modeling](https://mlsanity.com/paper/2506.14293/) (2025)
- [Exploring LLMs for South Asian Music Understanding and Generation](https://mlsanity.com/paper/2606.05522/) (2026)

## Abstract (arXiv)

Creating lyrics and melodies for the vocal track in a symbolic format, known as song composition, demands expert musical knowledge of melody, an advanced understanding of lyrics, and precise alignment between them. Despite achievements in sub-tasks such as lyric generation, lyric-to-melody, and melody-to-lyric, etc, a unified model for song composition has not yet been achieved. In this paper, we introduce SongComposer, a pioneering step towards a unified song composition model that can readily create symbolic lyrics and melodies following instructions. SongComposer is a music-specialized large language model (LLM) that, for the first time, integrates the capability of simultaneously composing lyrics and melodies into LLMs by leveraging three key innovations: 1) a flexible tuple format for word-level alignment of lyrics and melodies, 2) an extended tokenizer vocabulary for song notes, with scalar initialization based on musical knowledge to capture rhythm, and 3) a multi-stage pipeline that captures musical structure, starting with motif-level melody patterns and progressing to phrase-level structure for improved coherence. Extensive experiments demonstrate that SongComposer outperforms advanced LLMs, including GPT-4, in tasks such as lyric-to-melody generation, melody-to-lyric generation, song continuation, and text-to-song creation. Moreover, we will release SongCompose, a large-scale dataset for training, containing paired lyrics and melodies in Chinese and English.