# Recursive speech separation for unknown number of speakers

arXiv: https://arxiv.org/abs/1904.03065 · HTML: https://mlsanity.com/paper/1904.03065/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 14 of 14)

- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Latent Autoregressive Source Separation](https://mlsanity.com/paper/2301.08562/) (2023)
- [Online Neural Diarization of Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2206.02432/) (2022)
- [Learning to Separate Voices by Spatial Regions](https://mlsanity.com/paper/2207.04203/) (2022)
- [Disentangling the Impacts of Language and Channel Variability on Speech Separation Networks](https://mlsanity.com/paper/2203.16040/) (2022)
- [MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation](https://mlsanity.com/paper/2211.07302/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations](https://mlsanity.com/paper/2303.12002/) (2023)
- [Iterative Sound Source Localization for Unknown Number of Sources](https://mlsanity.com/paper/2206.12273/) (2022)
- [Unsupervised Speaker Diarization in Distributed IoT Networks Using Federated Learning](https://mlsanity.com/paper/2404.10842/) (2024)
- [LightSAFT: Lightweight Latent Source Aware Frequency Transform for Source Separation](https://mlsanity.com/paper/2111.12516/) (2021)
- [Coarse-to-Fine Recursive Speech Separation for Unknown Number of Speakers](https://mlsanity.com/paper/2203.16054/) (2022)
- [Speech Separation based on Contrastive Learning and Deep Modularization](https://mlsanity.com/paper/2305.10652/) (2023)
- [Improving Neural Diarization through Speaker Attribute Attractors and Local Dependency Modeling](https://mlsanity.com/paper/2506.05593/) (2025)

## Abstract (arXiv)

In this paper we propose a method of single-channel speaker-independent multi-speaker speech separation for an unknown number of speakers. As opposed to previous works, in which the number of speakers is assumed to be known in advance and speech separation models are specific for the number of speakers, our proposed method can be applied to cases with different numbers of speakers using a single model by recursively separating a speaker. To make the separation model recursively applicable, we propose one-and-rest permutation invariant training (OR-PIT). Evaluation on WSJ0-2mix and WSJ0-3mix datasets show that our proposed method achieves state-of-the-art results for two- and three-speaker mixtures with a single model. Moreover, the same model can separate four-speaker mixture, which was never seen during the training. We further propose the detection of the number of speakers in a mixture during recursive separation and show that this approach can more accurately estimate the number of speakers than detection in advance by using a deep neural network based classifier.