# Generation of lyrics lines conditioned on music audio clips

arXiv: https://arxiv.org/abs/2009.14375 · HTML: https://mlsanity.com/paper/2009.14375/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 3 of 3)

- [LyricJam: A system for generating lyrics for live instrumental music](https://mlsanity.com/paper/2106.01960/) (2021)
- [LyricJam Sonic: A Generative System for Real-Time Composition and Musical Improvisation](https://mlsanity.com/paper/2210.15638/) (2022)
- [Syllable-level lyrics generation from melody exploiting character-level language model](https://mlsanity.com/paper/2310.00863/) (2023)

## Abstract (arXiv)

We present a system for generating novel lyrics lines conditioned on music audio. A bimodal neural network model learns to generate lines conditioned on any given short audio clip. The model consists of a spectrogram variational autoencoder (VAE) and a text VAE. Both automatic and human evaluations demonstrate effectiveness of our model in generating lines that have an emotional impact matching a given audio clip. The system is intended to serve as a creativity tool for songwriters.