Controllable Lyrics-to-Melody Generation
Not on the map. Why?
Abstract (arXiv)
Lyrics-to-melody generation is an interesting and challenging topic in AI music research field. Due to the difficulty of learning the correlations between lyrics and melody, previous methods suffer from low generation quality and lack of controllability. Controllability of generative models enables human interaction with models to generate desired contents, which is especially important in music generation tasks towards human-centered AI that can facilitate musicians in creative activities. To address these issues, we propose a controllable lyrics-to-melody generation network, ConL2M, which is able to generate realistic melodies from lyrics in user-desired musical style. Our work contains three main novelties: 1) To model the dependencies of music attributes cross multiple sequences, inter-branch memory fusion (Memofu) is proposed to enable information flow between multi-branch stacked LSTM architecture; 2) Reference style embedding (RSE) is proposed to improve the quality of generation as well as control the musical style of generated melodies; 3) Sequence-level statistical loss (SeqLoss) is proposed to help the model learn sequence-level features of melodies given lyrics. Verified by evaluation metrics for music quality and controllability, initial study of controllable lyrics-to-melody generation shows better generation quality and the feasibility of interacting with users to generate the melodies in desired musical styles when given lyrics.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 5 Jun 2023.
Builds on
23 of 33 references.
References · 23
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2022Generative Adversarial Networkscited by 5,627
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
- 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
- 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
- 2018The relativistic discriminator: a key element missing from standard GANcited by 238
- 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
- 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
- 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
- 2020A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directionscited by 56
11 more
- 2017Deep Learning Techniques for Music Generation -- A Surveycited by 49
- 2020SongMASS: Automatic Song Writing with Pre-training and Alignment Constraintcited by 41
- 2020Learning Interpretable Representation for Controllable Polyphonic Music Generationcited by 40
- 2019Conditional LSTM-GAN for Melody Generation from Lyricscited by 31
- 2021Controllable deep melody generation via hierarchical music structure representationcited by 30
- 2020Music SketchNet: Controllable Music Generation via Factorized Representations of Pitch and Rhythmcited by 28
- 2021TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Methodcited by 20
- 2018Neural Melody Composition from Lyricscited by 10
- 2016Text-based LSTM networks for Automatic Music Compositioncited by 6
- 2016Algorithmic Songwriting with ALYSIAcited by 5
- 2022Interpretable Melody Generation from Lyrics with Discrete-Valued Adversarial Trainingcited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.