# Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

arXiv: https://arxiv.org/abs/2404.11275 · HTML: https://mlsanity.com/paper/2404.11275/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus](https://mlsanity.com/paper/2112.10358/) (2021)
- [Recognizing Multi-talker Speech with Permutation Invariant Training](https://mlsanity.com/paper/1704.01985/) (2017)
- [Streaming end-to-end multi-talker speech recognition](https://mlsanity.com/paper/2011.13148/) (2020)
- [End-to-End Monaural Multi-speaker ASR System without Pretraining](https://mlsanity.com/paper/1811.02062/) (2018)
- [Large-Scale Domain Adaptation via Teacher-Student Learning](https://mlsanity.com/paper/1708.05466/) (2017)
- [The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks](https://mlsanity.com/paper/2110.09958/) (2021)
- [Continuous Streaming Multi-Talker ASR with Dual-path Transducers](https://mlsanity.com/paper/2109.08555/) (2021)
- [Tackling the Cocktail Fork Problem for Separation and Transcription of Real-World Soundtracks](https://mlsanity.com/paper/2212.07327/) (2022)
- [Multi-Task Audio Source Separation](https://mlsanity.com/paper/2107.06467/) (2021)

## Built on by (most cited first, 1 of 1)

- [Robust Speech Activity Detection in the Presence of Singing Voice](https://mlsanity.com/paper/2512.09713/) (2025)

## Abstract (arXiv)

In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.