# Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music

arXiv: https://arxiv.org/abs/2204.03307 · HTML: https://mlsanity.com/paper/2204.03307/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [End-to-end Speech Recognition with Word-based RNN Language Models](https://mlsanity.com/paper/1808.02608/) (2018)
- [DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm](https://mlsanity.com/paper/1906.10606/) (2019)
- [End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model](https://mlsanity.com/paper/1902.06797/) (2019)
- [Low Resource Audio-to-Lyrics Alignment From Polyphonic Music Recordings](https://mlsanity.com/paper/2102.09202/) (2021)
- [Automatic Lyrics Transcription using Dilated Convolutional Neural Networks with Self-Attention](https://mlsanity.com/paper/2007.06486/) (2020)
- [Acoustic Modeling for Automatic Lyrics-to-Audio Alignment](https://mlsanity.com/paper/1906.10369/) (2019)
- [Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?](https://mlsanity.com/paper/1909.10200/) (2019)

## Built on by (most cited first, 7 of 7)

- [Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework](https://mlsanity.com/paper/2210.16798/) (2022)
- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Dynamic Transformers Provide a False Sense of Efficiency](https://mlsanity.com/paper/2305.12228/) (2023)
- [token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text](https://mlsanity.com/paper/2210.16755/) (2022)
- [Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic Singing Voice Understanding Tasks: Three Case Studies](https://mlsanity.com/paper/2306.12714/) (2023)
- [More than words: Advancements and challenges in speech recognition for singing](https://mlsanity.com/paper/2403.09298/) (2024)
- [Elucidate Gender Fairness in Singing Voice Transcription](https://mlsanity.com/paper/2308.02898/) (2023)

## Abstract (arXiv)

Lyrics transcription of polyphonic music is challenging not only because the singing vocals are corrupted by the background music, but also because the background music and the singing style vary across music genres, such as pop, metal, and hip hop, which affects lyrics intelligibility of the song in different ways. In this work, we propose to transcribe the lyrics of polyphonic music using a novel genre-conditioned network. The proposed network adopts pre-trained model parameters, and incorporates the genre adapters between layers to capture different genre peculiarities for lyrics-genre pairs, thereby only requiring lightweight genre-specific parameters for training. Our experiments show that the proposed genre-conditioned network outperforms the existing lyrics transcription systems.