# TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method

arXiv: https://arxiv.org/abs/2109.09617 · HTML: https://mlsanity.com/paper/2109.09617/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint](https://mlsanity.com/paper/2012.05168/) (2020)
- [Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering](https://mlsanity.com/paper/2004.11892/) (2020)
- [Conditional LSTM-GAN for Melody Generation from Lyrics](https://mlsanity.com/paper/1908.05551/) (2019)
- [SongNet: Rigid Formats Controlled Text Generation](https://mlsanity.com/paper/2004.08022/) (2020)
- [DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling](https://mlsanity.com/paper/2107.01875/) (2021)
- [PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music](https://mlsanity.com/paper/2010.08091/) (2020)
- [DopeLearning: A Computational Approach to Rap Lyrics Generation](https://mlsanity.com/paper/1505.04771/) (2015)
- [Neural Melody Composition from Lyrics](https://mlsanity.com/paper/1809.04318/) (2018)
- [Text-based LSTM networks for Automatic Music Composition](https://mlsanity.com/paper/1604.05358/) (2016)
- [On large-scale genre classification in symbolically encoded music by automatic identification of repeating patterns](https://mlsanity.com/paper/1910.09242/) (2019)
- [Algorithmic Songwriting with ALYSIA](https://mlsanity.com/paper/1612.01058/) (2016)

## Built on by (most cited first, 20 of 20)

- [Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation](https://mlsanity.com/paper/2210.10349/) (2022)
- [Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls](https://mlsanity.com/paper/2307.10304/) (2023)
- [MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models](https://mlsanity.com/paper/2310.11954/) (2023)
- [GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework](https://mlsanity.com/paper/2305.10841/) (2023)
- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)
- [Exploring the Collaborative Co-Creation Process with AI: A Case Study in Novice Music Production](https://mlsanity.com/paper/2501.15276/) (2025)
- [Regeneration Learning: A Learning Paradigm for Data Generation](https://mlsanity.com/paper/2301.08846/) (2023)
- [A Comprehensive Survey for Evaluation Methodologies of AI-Generated Music](https://mlsanity.com/paper/2308.13736/) (2023)
- [Motifs, Phrases, and Beyond: The Modelling of Structure in Symbolic Music Generation](https://mlsanity.com/paper/2403.07995/) (2024)
- [Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation](https://mlsanity.com/paper/2208.05697/) (2022)
- [Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics](https://mlsanity.com/paper/2303.15705/) (2023)
- [MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks](https://mlsanity.com/paper/2208.14345/) (2022)
- [WuYun: Exploring hierarchical skeleton-guided melody generation using knowledge-enhanced deep learning](https://mlsanity.com/paper/2301.04488/) (2023)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [Mamba-Diffusion Model with Learnable Wavelet for Controllable Symbolic Music Generation](https://mlsanity.com/paper/2505.03314/) (2025)
- [Controllable Lyrics-to-Melody Generation](https://mlsanity.com/paper/2306.02613/) (2023)
- [MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation](https://mlsanity.com/paper/2407.03188/) (2024)
- [CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls](https://mlsanity.com/paper/2412.09887/) (2024)
- [MusicAIR: A Multimodal AI Music Generation Framework Powered by an Algorithm-Driven Core](https://mlsanity.com/paper/2511.17323/) (2025)
- [Aligning Language Models for Lyric-to-Melody Generation with Rule-Based Musical Constraints](https://mlsanity.com/paper/2604.18489/) (2026)

## Abstract (arXiv)

Lyric-to-melody generation is an important task in automatic songwriting. Previous lyric-to-melody generation systems usually adopt end-to-end models that directly generate melodies from lyrics, which suffer from several issues: 1) lack of paired lyric-melody training data; 2) lack of control on generated melodies. In this paper, we develop TeleMelody, a two-stage lyric-to-melody generation system with music template (e.g., tonality, chord progression, rhythm pattern, and cadence) to bridge the gap between lyrics and melodies (i.e., the system consists of a lyric-to-template module and a template-to-melody module). TeleMelody has two advantages. First, it is data efficient. The template-to-melody module is trained in a self-supervised way (i.e., the source template is extracted from the target melody) that does not need any lyric-melody paired data. The lyric-to-template module is made up of some rules and a lyric-to-rhythm model, which is trained with paired lyric-rhythm data that is easier to obtain than paired lyric-melody data. Second, it is controllable. The design of template ensures that the generated melodies can be controlled by adjusting the musical elements in template. Both subjective and objective experimental evaluations demonstrate that TeleMelody generates melodies with higher quality, better controllability, and less requirement on paired lyric-melody data than previous generation systems.