# Improving Voice Separation by Incorporating End-to-end Speech Recognition

arXiv: https://arxiv.org/abs/1911.12928 · HTML: https://mlsanity.com/paper/1911.12928/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [End-to-end lyrics Recognition with Voice to Singing Style Transfer](https://mlsanity.com/paper/2102.08575/) (2021)
- [On permutation invariant training for speech source separation](https://mlsanity.com/paper/2102.04945/) (2021)
- [Iteratively Improving Speech Recognition and Voice Conversion](https://mlsanity.com/paper/2305.15055/) (2023)
- [Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extraction](https://mlsanity.com/paper/2503.08798/) (2025)
- [C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction](https://mlsanity.com/paper/2504.00750/) (2025)
- [VCSE: Time-Domain Visual-Contextual Speaker Extraction Network](https://mlsanity.com/paper/2210.06177/) (2022)

## Abstract (arXiv)

Despite recent advances in voice separation methods, many challenges remain in realistic scenarios such as noisy recording and the limits of available data. In this work, we propose to explicitly incorporate the phonetic and linguistic nature of speech by taking a transfer learning approach using an end-to-end automatic speech recognition (E2EASR) system. The voice separation is conditioned on deep features extracted from E2EASR to cover the long-term dependence of phonetic aspects. Experimental results on speech separation and enhancement task on the AVSpeech dataset show that the proposed method significantly improves the signal-to-distortion ratio over the baseline model and even outperforms an audio visual model, that utilizes visual information of lip movements.