# Voice Separation with an Unknown Number of Multiple Speakers

arXiv: https://arxiv.org/abs/2003.01531 · HTML: https://mlsanity.com/paper/2003.01531/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Feb 2020.

## Built on by (most cited first, 30 of 46)

- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation](https://mlsanity.com/paper/2403.18257/) (2024)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentions](https://mlsanity.com/paper/2302.11824/) (2023)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [SepIt: Approaching a Single Channel Speech Separation Bound](https://mlsanity.com/paper/2205.11801/) (2022)
- [Towards Neural Diarization for Unlimited Numbers of Speakers Using Global and Local Attractors](https://mlsanity.com/paper/2107.01545/) (2021)
- [Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation](https://mlsanity.com/paper/2301.10752/) (2023)
- [Many-Speakers Single Channel Speech Separation with Optimal Permutation Training](https://mlsanity.com/paper/2104.08955/) (2021)
- [Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation](https://mlsanity.com/paper/2103.00819/) (2021)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [OCD: Learning to Overfit with Conditional Diffusion Models](https://mlsanity.com/paper/2210.00471/) (2022)
- [Personalized Keyphrase Detection using Speaker and Environment Information](https://mlsanity.com/paper/2104.13970/) (2021)
- [Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data](https://mlsanity.com/paper/2105.04727/) (2021)
- [ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding](https://mlsanity.com/paper/2207.09514/) (2022)
- [ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement](https://mlsanity.com/paper/2212.11377/) (2022)
- [REAL-M: Towards Speech Separation on Real Mixtures](https://mlsanity.com/paper/2110.10812/) (2021)
- [MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training](https://mlsanity.com/paper/2202.03875/) (2022)
- [A Neural State-Space Model Approach to Efficient Speech Separation](https://mlsanity.com/paper/2305.16932/) (2023)
- [Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speech](https://mlsanity.com/paper/2112.05863/) (2021)
- [On permutation invariant training for speech source separation](https://mlsanity.com/paper/2102.04945/) (2021)
- [Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain](https://mlsanity.com/paper/2110.04791/) (2021)
- [Exploring Self-Attention Mechanisms for Speech Separation](https://mlsanity.com/paper/2202.02884/) (2022)
- [Learning to Separate Voices by Spatial Regions](https://mlsanity.com/paper/2207.04203/) (2022)
- [UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures](https://mlsanity.com/paper/2305.20054/) (2023)
- [MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation](https://mlsanity.com/paper/2211.07302/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Multi-Dimensional and Multi-Scale Modeling for Speech Separation Optimized by Discriminative Learning](https://mlsanity.com/paper/2303.03737/) (2023)

## Abstract (arXiv)

We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.