# Acoustic Modeling for Automatic Lyrics-to-Audio Alignment

arXiv: https://arxiv.org/abs/1906.10369 · HTML: https://mlsanity.com/paper/1906.10369/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2019.

## Built on by (most cited first, 5 of 5)

- [Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages](https://mlsanity.com/paper/2306.07744/) (2023)
- [Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music](https://mlsanity.com/paper/2204.03307/) (2022)
- [End-to-end lyrics Recognition with Voice to Singing Style Transfer](https://mlsanity.com/paper/2102.08575/) (2021)
- [And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?](https://mlsanity.com/paper/2210.01256/) (2022)
- [More than words: Advancements and challenges in speech recognition for singing](https://mlsanity.com/paper/2403.09298/) (2024)

## Abstract (arXiv)

Automatic lyrics to polyphonic audio alignment is a challenging task not only because the vocals are corrupted by background music, but also there is a lack of annotated polyphonic corpus for effective acoustic modeling. In this work, we propose (1) using additional speech and music-informed features and (2) adapting the acoustic models trained on a large amount of solo singing vocals towards polyphonic music using a small amount of in-domain data. Incorporating additional information such as voicing and auditory features together with conventional acoustic features aims to bring robustness against the increased spectro-temporal variations in singing vocals. By adapting the acoustic model using a small amount of polyphonic audio data, we reduce the domain mismatch between training and testing data. We perform several alignment experiments and present an in-depth alignment error analysis on acoustic features, and model adaptation techniques. The results demonstrate that the proposed strategy provides a significant error reduction of word boundary alignment over comparable existing systems, especially on more challenging polyphonic data with long-duration musical interludes.