# Syllable-level lyrics generation from melody exploiting character-level language model

arXiv: https://arxiv.org/abs/2310.00863 · HTML: https://mlsanity.com/paper/2310.00863/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Conditional LSTM-GAN for Melody Generation from Lyrics](https://mlsanity.com/paper/1908.05551/) (2019)
- [Melody-Conditioned Lyrics Generation with SeqGANs](https://mlsanity.com/paper/2010.14709/) (2020)
- [ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships](https://mlsanity.com/paper/2207.05688/) (2022)
- [Youling: an AI-Assisted Lyrics Creation System](https://mlsanity.com/paper/2201.06724/) (2022)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [A Hierarchical Attention Based Seq2seq Model for Chinese Lyrics Generation](https://mlsanity.com/paper/1906.06481/) (2019)
- [A Syllable-Structured, Contextually-Based Conditionally Generation of Chinese Lyrics](https://mlsanity.com/paper/1906.09322/) (2019)
- [Theme-aware generation model for chinese lyrics](https://mlsanity.com/paper/1906.02134/) (2019)
- [Generation of lyrics lines conditioned on music audio clips](https://mlsanity.com/paper/2009.14375/) (2020)

## Built on by (most cited first, 2 of 2)

- [Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control](https://mlsanity.com/paper/2411.13100/) (2024)
- [CoLyricist: Enhancing Lyric Writing with AI through Workflow-Aligned Support](https://mlsanity.com/paper/2602.22606/) (2026)

## Abstract (arXiv)

The generation of lyrics tightly connected to accompanying melodies involves establishing a mapping between musical notes and syllables of lyrics. This process requires a deep understanding of music constraints and semantic patterns at syllable-level, word-level, and sentence-level semantic meanings. However, pre-trained language models specifically designed at the syllable level are publicly unavailable. To solve these challenging issues, we propose to exploit fine-tuning character-level language models for syllable-level lyrics generation from symbolic melody. In particular, our method endeavors to incorporate linguistic knowledge of the language model into the beam search process of a syllable-level Transformer generator network. Additionally, by exploring ChatGPT-based evaluation for generated lyrics, along with human subjective evaluation, we demonstrate that our approach enhances the coherence and correctness of the generated lyrics, eliminating the need to train expensive new language models.