# Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation

arXiv: https://arxiv.org/abs/2208.05697 · HTML: https://mlsanity.com/paper/2208.05697/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018)
- [MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2106.05630/) (2021)
- [Analogical Reasoning on Chinese Morphological and Semantic Relations](https://mlsanity.com/paper/1805.06504/) (2018)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Conditional LSTM-GAN for Melody Generation from Lyrics](https://mlsanity.com/paper/1908.05551/) (2019)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling](https://mlsanity.com/paper/2107.01875/) (2021)
- [PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music](https://mlsanity.com/paper/2010.08091/) (2020)
- [TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method](https://mlsanity.com/paper/2109.09617/) (2021)
- [DopeLearning: A Computational Approach to Rap Lyrics Generation](https://mlsanity.com/paper/1505.04771/) (2015)
- [Neural Melody Composition from Lyrics](https://mlsanity.com/paper/1809.04318/) (2018)
- [A variational autoencoder for music generation controlled by tonal tension](https://mlsanity.com/paper/2010.06230/) (2020)

## Built on by (most cited first, 6 of 6)

- [Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use](https://mlsanity.com/paper/2312.04455/) (2023)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework](https://mlsanity.com/paper/2305.10841/) (2023)
- [MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks](https://mlsanity.com/paper/2208.14345/) (2022)
- [SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training](https://mlsanity.com/paper/2412.18107/) (2024)
- [SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away](https://mlsanity.com/paper/2602.24071/) (2026)

## Abstract (arXiv)

Lyric-to-melody generation is an important task in songwriting, and is also quite challenging due to its unique characteristics: the generated melodies should not only follow good musical patterns, but also align with features in lyrics such as rhythms and structures. These characteristics cannot be well handled by neural generation models that learn lyric-to-melody mapping in an end-to-end way, due to several issues: (1) lack of aligned lyric-melody training data to sufficiently learn lyric-melody feature alignment; (2) lack of controllability in generation to better and explicitly align the lyric-melody features. In this paper, we propose Re-creation of Creations (ROC), a new paradigm for lyric-to-melody generation. ROC generates melodies according to given lyrics and also conditions on user-designated chord progression. It addresses the above issues through a generation-retrieval pipeline. Specifically, our paradigm has two stages: (1) creation stage, where a huge amount of music fragments generated by a neural melody language model are indexed in a database through several key features (e.g., chords, tonality, rhythm, and structural information); (2) re-creation stage, where melodies are re-created by retrieving music fragments from the database according to the key features from lyrics and concatenating best music fragments based on composition guidelines and melody language model scores. ROC has several advantages: (1) It only needs unpaired melody data to train melody language model, instead of paired lyric-melody data in previous models. (2) It achieves good lyric-melody feature alignment in lyric-to-melody generation. Tested by English and Chinese lyrics, ROC outperforms previous neural based lyric-to-melody generation models on both objective and subjective metrics.