# MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription

arXiv: https://arxiv.org/abs/2108.02625 · HTML: https://mlsanity.com/paper/2108.02625/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 10 of 10)

- [DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm](https://mlsanity.com/paper/1906.10606/) (2019)
- [End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model](https://mlsanity.com/paper/1902.06797/) (2019)
- [Multistream CNN for Robust Acoustic Modeling](https://mlsanity.com/paper/2005.10470/) (2020)
- [State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions](https://mlsanity.com/paper/1910.00716/) (2019)
- [ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition](https://mlsanity.com/paper/2005.10469/) (2020)
- [Low Resource Audio-to-Lyrics Alignment From Polyphonic Music Recordings](https://mlsanity.com/paper/2102.09202/) (2021)
- [Automatic Lyrics Transcription using Dilated Convolutional Neural Networks with Self-Attention](https://mlsanity.com/paper/2007.06486/) (2020)
- [Data Cleansing with Contrastive Learning for Vocal Note Event Annotations](https://mlsanity.com/paper/2008.02069/) (2020)
- [Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?](https://mlsanity.com/paper/1909.10200/) (2019)
- [Computational Pronunciation Analysis in Sung Utterances](https://mlsanity.com/paper/2106.10977/) (2021)

## Built on by (most cited first, 7 of 7)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Lyrics Transcription for Humans: A Readability-Aware Benchmark](https://mlsanity.com/paper/2408.06370/) (2024)
- [Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark](https://mlsanity.com/paper/2311.13987/) (2023)
- [Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic Singing Voice Understanding Tasks: Three Case Studies](https://mlsanity.com/paper/2306.12714/) (2023)
- [More than words: Advancements and challenges in speech recognition for singing](https://mlsanity.com/paper/2403.09298/) (2024)
- [Elucidate Gender Fairness in Singing Voice Transcription](https://mlsanity.com/paper/2308.02898/) (2023)

## Abstract (arXiv)

This paper makes several contributions to automatic lyrics transcription (ALT) research. Our main contribution is a novel variant of the Multistreaming Time-Delay Neural Network (MTDNN) architecture, called MSTRE-Net, which processes the temporal information using multiple streams in parallel with varying resolutions keeping the network more compact, and thus with a faster inference and an improved recognition rate than having identical TDNN streams. In addition, two novel preprocessing steps prior to training the acoustic model are proposed. First, we suggest using recordings from both monophonic and polyphonic domains during training the acoustic model. Second, we tag monophonic and polyphonic recordings with distinct labels for discriminating non-vocal silence and music instances during alignment. Moreover, we present a new test set with a considerably larger size and a higher musical variability compared to the existing datasets used in ALT literature, while maintaining the gender balance of the singers. Our best performing model sets the state-of-the-art in lyrics transcription by a large margin. For reproducibility, we publicly share the identifiers to retrieve the data used in this paper.