# Conditional LSTM-GAN for Melody Generation from Lyrics

arXiv: https://arxiv.org/abs/1908.05551 · HTML: https://mlsanity.com/paper/1908.05551/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2019.

## Built on by (most cited first, 30 of 31)

- [JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models](https://mlsanity.com/paper/2308.04729/) (2023)
- [TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Method](https://mlsanity.com/paper/2109.09617/) (2021)
- [Text2midi: Generating Symbolic Music from Captions](https://mlsanity.com/paper/2412.16526/) (2024)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation](https://mlsanity.com/paper/2310.19180/) (2023)
- [GAN Computers Generate Arts? A Survey on Visual Arts, Music, and Literary Text Generation using Generative Adversarial Network](https://mlsanity.com/paper/2108.03857/) (2021)
- [Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompaniment](https://mlsanity.com/paper/2404.09313/) (2024)
- [A Review of Intelligent Music Generation Systems](https://mlsanity.com/paper/2211.09124/) (2022)
- [Exploring The Effect of High-frequency Components in GANs Training](https://mlsanity.com/paper/2103.11093/) (2021)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition](https://mlsanity.com/paper/2402.17645/) (2024)
- [Unsupervised Melody-to-Lyric Generation](https://mlsanity.com/paper/2305.19228/) (2023)
- [Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation](https://mlsanity.com/paper/2208.05697/) (2022)
- [Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics](https://mlsanity.com/paper/2303.15705/) (2023)
- [GANmapper: geographical data translation](https://mlsanity.com/paper/2108.04232/) (2021)
- [MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks](https://mlsanity.com/paper/2208.14345/) (2022)
- [CLoG: Benchmarking Continual Learning of Image Generation Models](https://mlsanity.com/paper/2406.04584/) (2024)
- [Text-driven Video Prediction](https://mlsanity.com/paper/2210.02872/) (2022)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [Generating Privacy-Preserving Process Data with Deep Generative Models](https://mlsanity.com/paper/2203.07949/) (2022)
- [Controllable Lyrics-to-Melody Generation](https://mlsanity.com/paper/2306.02613/) (2023)
- [Syllable-level lyrics generation from melody exploiting character-level language model](https://mlsanity.com/paper/2310.00863/) (2023)
- [Joint Learning of Wording and Formatting for Singable Melody-to-Lyric Generation](https://mlsanity.com/paper/2307.02146/) (2023)
- [Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control](https://mlsanity.com/paper/2411.13100/) (2024)
- [Conditional Generative Adversarial Network for keystroke presentation attack](https://mlsanity.com/paper/2212.08445/) (2022)
- [Deep Attention-Based Alignment Network for Melody Generation from Incomplete Lyrics](https://mlsanity.com/paper/2301.10015/) (2023)
- [Unsupervised Melody-Guided Lyrics Generation](https://mlsanity.com/paper/2305.07760/) (2023)
- [Tuning Into Bias: A Computational Study of Gender Bias in Song Lyrics](https://mlsanity.com/paper/2409.15949/) (2024)
- [SongSong: A Time Phonograph for Chinese SongCi Music from Thousand of Years Away](https://mlsanity.com/paper/2602.24071/) (2026)
- [Fusing Memory and Attention: A study on LSTM, Transformer and Hybrid Architectures for Symbolic Music Generation](https://mlsanity.com/paper/2603.21282/) (2026)

## Abstract (arXiv)

Melody generation from lyrics has been a challenging research issue in the field of artificial intelligence and music, which enables to learn and discover latent relationship between interesting lyrics and accompanying melody. Unfortunately, the limited availability of paired lyrics-melody dataset with alignment information has hindered the research progress. To address this problem, we create a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment through leveraging different music sources where alignment relationship between syllables and music attributes is extracted. Most importantly, we propose a novel deep generative model, conditional Long Short-Term Memory - Generative Adversarial Network (LSTM-GAN) for melody generation from lyrics, which contains a deep LSTM generator and a deep LSTM discriminator both conditioned on lyrics. In particular, lyrics-conditioned melody and alignment relationship between syllables of given lyrics and notes of predicted melody are generated simultaneously. Experimental results have proved the effectiveness of our proposed lyrics-to-melody generative model, where plausible and tuneful sequences can be inferred from lyrics.