# Single microphone speaker extraction using unified time-frequency Siamese-Unet

arXiv: https://arxiv.org/abs/2203.02941 · HTML: https://mlsanity.com/paper/2203.02941/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 6 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [Single channel voice separation for unknown number of speakers under reverberant and noisy settings](https://mlsanity.com/paper/2011.02329/) (2020)
- [X-TaSNet: Robust and Accurate Time-Domain Speaker Extraction Network](https://mlsanity.com/paper/2010.12766/) (2020)
- [Deep Extractor Network for Target Speaker Recovery From Single Channel Speech Mixtures](https://mlsanity.com/paper/1807.08974/) (2018)
- [Atss-Net: Target Speaker Separation via Attention-based Neural Network](https://mlsanity.com/paper/2005.09200/) (2020)
- [Time-domain speaker extraction network](https://mlsanity.com/paper/2004.14762/) (2020)
- [Robust Speaker Extraction Network Based on Iterative Refined Adaptation](https://mlsanity.com/paper/2011.02102/) (2020)
- [Attention-based scaling adaptation for target speech extraction](https://mlsanity.com/paper/2010.10923/) (2020)

## Built on by (most cited first, 1 of 1)

- [End-to-End Multi-Microphone Speaker Extraction Using Relative Transfer Functions](https://mlsanity.com/paper/2502.06285/) (2025)

## Abstract (arXiv)

In this paper we present a unified time-frequency method for speaker extraction in clean and noisy conditions. Given a mixed signal, along with a reference signal, the common approaches for extracting the desired speaker are either applied in the time-domain or in the frequency-domain. In our approach, we propose a Siamese-Unet architecture that uses both representations. The Siamese encoders are applied in the frequency-domain to infer the embedding of the noisy and reference spectra, respectively. The concatenated representations are then fed into the decoder to estimate the real and imaginary components of the desired speaker, which are then inverse-transformed to the time-domain. The model is trained with the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) loss to exploit the time-domain information. The time-domain loss is also regularized with frequency-domain loss to preserve the speech patterns. Experimental results demonstrate that the unified approach is not only very easy to train, but also provides superior results as compared with state-of-the-art (SOTA) Blind Source Separation (BSS) methods, as well as commonly used speaker extraction approach.