# LyricJam: A system for generating lyrics for live instrumental music

arXiv: https://arxiv.org/abs/2106.01960 · HTML: https://mlsanity.com/paper/2106.01960/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Narrative Planning: Balancing Plot and Character](https://mlsanity.com/paper/1401.3841/) (2014)
- [DopeLearning: A Computational Approach to Rap Lyrics Generation](https://mlsanity.com/paper/1505.04771/) (2015)
- [Cue Me In: Content-Inducing Approaches to Interactive Story Generation](https://mlsanity.com/paper/2010.09935/) (2020)
- [Generating Thematic Chinese Poetry using Conditional Variational Autoencoders with Hybrid Decoders](https://mlsanity.com/paper/1711.07632/) (2017)
- [Guess who? Multilingual approach for the automated generation of author-stylized poetry](https://mlsanity.com/paper/1807.07147/) (2018)
- [Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval](https://mlsanity.com/paper/1711.08976/) (2017)
- [Algorithmic Songwriting with ALYSIA](https://mlsanity.com/paper/1612.01058/) (2016)
- [Shimon the Rapper: A Real-Time System for Human-Robot Interactive Rap Battles](https://mlsanity.com/paper/2009.09234/) (2020)
- [Generating lyrics with variational autoencoder and multi-modal artist embeddings](https://mlsanity.com/paper/1812.08318/) (2018)
- [Generation of lyrics lines conditioned on music audio clips](https://mlsanity.com/paper/2009.14375/) (2020)
- [The Evolution of Popular Music: USA 1960-2010](https://mlsanity.com/paper/1502.05417/) (2015)

## Built on by (most cited first, 3 of 3)

- [LyricJam Sonic: A Generative System for Real-Time Composition and Musical Improvisation](https://mlsanity.com/paper/2210.15638/) (2022)
- [CoLyricist: Enhancing Lyric Writing with AI through Workflow-Aligned Support](https://mlsanity.com/paper/2602.22606/) (2026)
- [Between Predictability and Randomness: Seeking Artistic Inspiration from AI Generative Models](https://mlsanity.com/paper/2506.12634/) (2025)

## Abstract (arXiv)

We describe a real-time system that receives a live audio stream from a jam session and generates lyric lines that are congruent with the live music being played. Two novel approaches are proposed to align the learned latent spaces of audio and text representations that allow the system to generate novel lyric lines matching live instrumental music. One approach is based on adversarial alignment of latent representations of audio and lyrics, while the other approach learns to transfer the topology from the music latent space to the lyric latent space. A user study with music artists using the system showed that the system was useful not only in lyric composition, but also encouraged the artists to improvise and find new musical expressions. Another user study demonstrated that users preferred the lines generated using the proposed methods to the lines generated by a baseline model.