# SpEx: Multi-Scale Time Domain Speaker Extraction Network

arXiv: https://arxiv.org/abs/2004.08326 · HTML: https://mlsanity.com/paper/2004.08326/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2020.

## Built on by (most cited first, 30 of 30)

- [FullSubNet+: Channel Attention FullSubNet with Complex Spectrograms for Speech Enhancement](https://mlsanity.com/paper/2203.12188/) (2022)
- [ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models](https://mlsanity.com/paper/2401.17230/) (2024)
- [An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits](https://mlsanity.com/paper/2212.10744/) (2022)
- [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025)
- [DDTSE: Discriminative Diffusion Model for Target Speech Extraction](https://mlsanity.com/paper/2309.13874/) (2023)
- [Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2404.12725/) (2024)
- [X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion](https://mlsanity.com/paper/2303.05023/) (2023)
- [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023)
- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction](https://mlsanity.com/paper/2312.10305/) (2023)
- [Multi-user VoiceFilter-Lite via Attentive Speaker Embedding](https://mlsanity.com/paper/2107.01201/) (2021)
- [Conditional Diffusion Model for Target Speaker Extraction](https://mlsanity.com/paper/2310.04791/) (2023)
- [Cross-attention Inspired Selective State Space Models for Target Sound Extraction](https://mlsanity.com/paper/2409.04803/) (2024)
- [WASE: Learning When to Attend for Speaker Extraction in Cocktail Party Environments](https://mlsanity.com/paper/2106.07016/) (2021)
- [Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction](https://mlsanity.com/paper/2505.20635/) (2025)
- [Closing the Gap between Single-User and Multi-User VoiceFilter-Lite](https://mlsanity.com/paper/2202.12169/) (2022)
- [Single microphone speaker extraction using unified time-frequency Siamese-Unet](https://mlsanity.com/paper/2203.02941/) (2022)
- [ACA-Net: Towards Lightweight Speaker Verification using Asymmetric Cross Attention](https://mlsanity.com/paper/2305.12121/) (2023)
- [TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion](https://mlsanity.com/paper/2401.14185/) (2024)
- [Personalized Speech Enhancement Without a Separate Speaker Embedding Model](https://mlsanity.com/paper/2406.09928/) (2024)
- [Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments](https://mlsanity.com/paper/2502.16611/) (2025)
- [GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model](https://mlsanity.com/paper/2512.20978/) (2025)
- [SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarios](https://mlsanity.com/paper/2607.02343/) (2026)
- [Sparsely Overlapped Speech Training in the Time Domain: Joint Learning of Target Speech Separation and Personal VAD Benefits](https://mlsanity.com/paper/2106.14371/) (2021)
- [Coarse-to-Fine Recursive Speech Separation for Unknown Number of Speakers](https://mlsanity.com/paper/2203.16054/) (2022)
- [VCSE: Time-Domain Visual-Contextual Speaker Extraction Network](https://mlsanity.com/paper/2210.06177/) (2022)
- [Improving Target Speaker Extraction with Sparse LDA-transformed Speaker Embeddings](https://mlsanity.com/paper/2301.06277/) (2023)
- [NeuroSpex: Neuro-Guided Speaker Extraction with Cross-Modal Attention](https://mlsanity.com/paper/2409.02489/) (2024)
- [Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling](https://mlsanity.com/paper/2510.26825/) (2025)
- [Contrastive Knowledge Distillation for Embedding Refinement in Personalized Speech Enhancement](https://mlsanity.com/paper/2601.16235/) (2026)

## Abstract (arXiv)

Speaker extraction aims to mimic humans' selective auditory attention by extracting a target speaker's voice from a multi-talker environment. It is common to perform the extraction in frequency-domain, and reconstruct the time-domain signal from the extracted magnitude and estimated phase spectra. However, such an approach is adversely affected by the inherent difficulty of phase estimation. Inspired by Conv-TasNet, we propose a time-domain speaker extraction network (SpEx) that converts the mixture speech into multi-scale embedding coefficients instead of decomposing the speech signal into magnitude and phase spectra. In this way, we avoid phase estimation. The SpEx network consists of four network components, namely speaker encoder, speech encoder, speaker extractor, and speech decoder. Specifically, the speech encoder converts the mixture speech into multi-scale embedding coefficients, the speaker encoder learns to represent the target speaker with a speaker embedding. The speaker extractor takes the multi-scale embedding coefficients and target speaker embedding as input and estimates a receptive mask. Finally, the speech decoder reconstructs the target speaker's speech from the masked embedding coefficients. We also propose a multi-task learning framework and a multi-scale embedding implementation. Experimental results show that the proposed SpEx achieves 37.3%, 37.7% and 15.0% relative improvements over the best baseline in terms of signal-to-distortion ratio (SDR), scale-invariant SDR (SI-SDR), and perceptual evaluation of speech quality (PESQ) under an open evaluation condition.