# Many-Speakers Single Channel Speech Separation with Optimal Permutation Training

arXiv: https://arxiv.org/abs/2104.08955 · HTML: https://mlsanity.com/paper/2104.08955/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020)
- [Single-Channel Multi-Speaker Separation using Deep Clustering](https://mlsanity.com/paper/1607.02173/) (2016)
- [Deep attractor network for single-microphone speaker separation](https://mlsanity.com/paper/1611.08930/) (2016)
- [FaSNet: Low-latency Adaptive Beamforming for Multi-microphone Audio Processing](https://mlsanity.com/paper/1909.13387/) (2019)
- [Speech Dereverberation Using Fully Convolutional Networks](https://mlsanity.com/paper/1803.08243/) (2018)
- [Deep Learning Based Phase Reconstruction for Speaker Separation: A Trigonometric Perspective](https://mlsanity.com/paper/1811.09010/) (2018)
- [Probabilistic Permutation Invariant Training for Speech Separation](https://mlsanity.com/paper/1908.01768/) (2019)
- [End-to-End Speech Separation with Unfolded Iterative Phase Reconstruction](https://mlsanity.com/paper/1804.10204/) (2018)
- [Towards Listening to 10 People Simultaneously: An Efficient Permutation Invariant Training of Audio Source Separation Using Sinkhorn's Algorithm](https://mlsanity.com/paper/2010.11871/) (2020)

## Built on by (most cited first, 11 of 11)

- [SepIt: Approaching a Single Channel Speech Separation Bound](https://mlsanity.com/paper/2205.11801/) (2022)
- [Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation](https://mlsanity.com/paper/2301.10752/) (2023)
- [Latent Autoregressive Source Separation](https://mlsanity.com/paper/2301.08562/) (2023)
- [OCD: Learning to Overfit with Conditional Diffusion Models](https://mlsanity.com/paper/2210.00471/) (2022)
- [Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience](https://mlsanity.com/paper/2402.03710/) (2024)
- [Learning to Separate Voices by Spatial Regions](https://mlsanity.com/paper/2207.04203/) (2022)
- [Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio](https://mlsanity.com/paper/2211.07951/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Multiple Choice Learning for Efficient Speech Separation with Many Speakers](https://mlsanity.com/paper/2411.18497/) (2024)
- [AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling](https://mlsanity.com/paper/2406.11619/) (2024)
- [Speech Separation based on Contrastive Learning and Deep Modularization](https://mlsanity.com/paper/2305.10652/) (2023)

## Abstract (arXiv)

Single channel speech separation has experienced great progress in the last few years. However, training neural speech separation for a large number of speakers (e.g., more than 10 speakers) is out of reach for the current methods, which rely on the Permutation Invariant Loss (PIT). In this work, we present a permutation invariant training that employs the Hungarian algorithm in order to train with an O(C3) time complexity, where C is the number of speakers, in comparison to O(C!) of PIT based methods. Furthermore, we present a modified architecture that can handle the increased number of speakers. Our approach separates up to 20 speakers and improves the previous results for large C by a wide margin.