# More than words: Advancements and challenges in speech recognition for singing

arXiv: https://arxiv.org/abs/2403.09298 · HTML: https://mlsanity.com/paper/2403.09298/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 59 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 14 of 14)

- [DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm](https://mlsanity.com/paper/1906.10606/) (2019)
- [End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model](https://mlsanity.com/paper/1902.06797/) (2019)
- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Transfer Learning of wav2vec 2.0 for Automatic Lyric Transcription](https://mlsanity.com/paper/2207.09747/) (2022)
- [Low Resource Audio-to-Lyrics Alignment From Polyphonic Music Recordings](https://mlsanity.com/paper/2102.09202/) (2021)
- [vocadito: A dataset of solo vocals with f0, note, and lyric annotations](https://mlsanity.com/paper/2110.05580/) (2021)
- [Automatic Lyrics Transcription using Dilated Convolutional Neural Networks with Self-Attention](https://mlsanity.com/paper/2007.06486/) (2020)
- [MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription](https://mlsanity.com/paper/2108.02625/) (2021)
- [Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music](https://mlsanity.com/paper/2204.03307/) (2022)
- [Acoustic Modeling for Automatic Lyrics-to-Audio Alignment](https://mlsanity.com/paper/1906.10369/) (2019)
- [PDAugment: Data Augmentation by Pitch and Duration Adjustments for Automatic Lyrics Transcription](https://mlsanity.com/paper/2109.07940/) (2021)
- [Transcribing Lyrics From Commercial Song Audio: The First Step Towards Singing Content Processing](https://mlsanity.com/paper/1804.05306/) (2018)
- [Listen, Read, and Identify: Multimodal Singing Language Identification of Music](https://mlsanity.com/paper/2103.01893/) (2021)
- [Singing Language Identification using a Deep Phonotactic Approach](https://mlsanity.com/paper/2105.15014/) (2021)

## Built on by (most cited first, 1 of 1)

- [Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation](https://mlsanity.com/paper/2609.11302/) (2026)

## Abstract (arXiv)

This paper addresses the challenges and advancements in speech recognition for singing, a domain distinctly different from standard speech recognition. Singing encompasses unique challenges, including extensive pitch variations, diverse vocal styles, and background music interference. We explore key areas such as phoneme recognition, language identification in songs, keyword spotting, and full lyrics transcription. I will describe some of my own experiences when performing research on these tasks just as they were starting to gain traction, but will also show how recent developments in deep learning and large-scale datasets have propelled progress in this field. My goal is to illuminate the complexities of applying speech recognition to singing, evaluate current capabilities, and outline future research directions.