# Towards Building an End-to-End Multilingual Automatic Lyrics Transcription Model

arXiv: https://arxiv.org/abs/2406.17618 · HTML: https://mlsanity.com/paper/2406.17618/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Multilingual Speech Recognition With A Single End-To-End Model](https://mlsanity.com/paper/1711.01694/) (2017)
- [Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languages](https://mlsanity.com/paper/1806.05059/) (2018)
- [End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model](https://mlsanity.com/paper/1902.06797/) (2019)
- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Transfer Learning of wav2vec 2.0 for Automatic Lyric Transcription](https://mlsanity.com/paper/2207.09747/) (2022)
- [Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages](https://mlsanity.com/paper/2306.07744/) (2023)
- [Low Resource Audio-to-Lyrics Alignment From Polyphonic Music Recordings](https://mlsanity.com/paper/2102.09202/) (2021)
- [PoLyScriber: Integrated Fine-tuning of Extractor and Lyrics Transcriber for Polyphonic Music](https://mlsanity.com/paper/2207.07336/) (2022)
- [Adapting pretrained speech model for Mandarin lyrics transcription and alignment](https://mlsanity.com/paper/2311.12488/) (2023)
- [Computational Pronunciation Analysis in Sung Utterances](https://mlsanity.com/paper/2106.10977/) (2021)

## Built on by (most cited first, 1 of 1)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)

## Abstract (arXiv)

Multilingual automatic lyrics transcription (ALT) is a challenging task due to the limited availability of labelled data and the challenges introduced by singing, compared to multilingual automatic speech recognition. Although some multilingual singing datasets have been released recently, English continues to dominate these collections. Multilingual ALT remains underexplored due to the scale of data and annotation quality. In this paper, we aim to create a multilingual ALT system with available datasets. Inspired by architectures that have been proven effective for English ALT, we adapt these techniques to the multilingual scenario by expanding the target vocabulary set. We then evaluate the performance of the multilingual model in comparison to its monolingual counterparts. Additionally, we explore various conditioning methods to incorporate language information into the model. We apply analysis by language and combine it with the language classification performance. Our findings reveal that the multilingual model performs consistently better than the monolingual models trained on the language subsets. Furthermore, we demonstrate that incorporating language information significantly enhances performance.