# End-to-end lyrics Recognition with Voice to Singing Style Transfer

arXiv: https://arxiv.org/abs/2102.08575 · HTML: https://mlsanity.com/paper/2102.08575/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model](https://mlsanity.com/paper/1902.06797/) (2019)
- [Improving Voice Separation by Incorporating End-to-end Speech Recognition](https://mlsanity.com/paper/1911.12928/) (2019)
- [Acoustic Modeling for Automatic Lyrics-to-Audio Alignment](https://mlsanity.com/paper/1906.10369/) (2019)
- [Speech-to-Singing Conversion in an Encoder-Decoder Framework](https://mlsanity.com/paper/2002.06595/) (2020)

## Built on by (most cited first, 5 of 5)

- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Leveraging Symmetrical Convolutional Transformer Networks for Speech to Singing Voice Style Transfer](https://mlsanity.com/paper/2208.12410/) (2022)
- [Iteratively Improving Speech Recognition and Voice Conversion](https://mlsanity.com/paper/2305.15055/) (2023)
- [Toward Leveraging Pre-Trained Self-Supervised Frontends for Automatic Singing Voice Understanding Tasks: Three Case Studies](https://mlsanity.com/paper/2306.12714/) (2023)
- [Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation](https://mlsanity.com/paper/2609.11302/) (2026)

## Abstract (arXiv)

Automatic transcription of monophonic/polyphonic music is a challenging task due to the lack of availability of large amounts of transcribed data. In this paper, we propose a data augmentation method that converts natural speech to singing voice based on vocoder based speech synthesizer. This approach, called voice to singing (V2S), performs the voice style conversion by modulating the F0 contour of the natural speech with that of a singing voice. The V2S model based style transfer can generate good quality singing voice thereby enabling the conversion of large corpora of natural speech to singing voice that is useful in building an E2E lyrics transcription system. In our experiments on monophonic singing voice data, the V2S style transfer provides a significant gain (relative improvements of 21%) for the E2E lyrics transcription system. We also discuss additional components like transfer learning and lyrics based language modeling to improve the performance of the lyrics transcription system.