Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation
On the map: Speech and audio language models → Vocoders and speech synthesis
Abstract (arXiv)
In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.
Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 20 of its 37 references that are in the map. First posted on arXiv 17 Apr 2024.
Builds on
20 of 37 references.
Closest ancestors · 2
Same direction, newest first
- 2022Tackling the Cocktail Fork Problem for Separation and Transcription of Real-World Soundtracks14 Dec
- 2021Multi-Task Audio Source Separation14 Jul
Other references · 18
- 2017Attention Is All You Needcited by 61,180
- 2014Adam: A Method for Stochastic Optimizationcited by 38,210
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 2,293
- 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 596
- 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 497
- 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 281
- 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 275
- 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 187
- 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 177
- 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 131
- 2020Continuous speech separation: dataset and analysiscited by 124
- 2021Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpuscited by 63
6 more
- 2017Recognizing Multi-talker Speech with Permutation Invariant Trainingcited by 46
- 2020Streaming end-to-end multi-talker speech recognitioncited by 41
- 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 38
- 2017Large-Scale Domain Adaptation via Teacher-Student Learningcited by 33
- 2021The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtrackscited by 24
- 2021Continuous Streaming Multi-Talker ASR with Dual-path Transducerscited by 19
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2021The Use of Voice Source Features for Sung Speech Recognition
- 2026MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Model
- 2025Exploiting Music Source Separation for Automatic Lyrics Transcription with Whisper
- 2023Iteratively Improving Speech Recognition and Voice Conversion
- 2024Task-Aware Unified Source Separation
- 2024More than words: Advancements and challenges in speech recognition for singing
- 2021Do You Listen with One or Two Microphones? A Unified ASR Model for Single and Multi-Channel Audio
- 2026Using Songs to Improve Kazakh Automatic Speech Recognition