# On permutation invariant training for speech source separation

arXiv: https://arxiv.org/abs/2102.04945 · HTML: https://mlsanity.com/paper/2102.04945/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 28 of 28)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020)
- [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019)
- [Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation](https://mlsanity.com/paper/1904.11148/) (2019)
- [Two-Step Sound Source Separation: Training on Learned Latent Targets](https://mlsanity.com/paper/1910.09804/) (2019)
- [Improving Universal Sound Separation Using Sound Classification](https://mlsanity.com/paper/1911.07951/) (2019)
- [Filterbank design for end-to-end speech separation](https://mlsanity.com/paper/1910.10400/) (2019)
- [An empirical study of Conv-TasNet](https://mlsanity.com/paper/2002.08688/) (2020)
- [Probabilistic Permutation Invariant Training for Speech Separation](https://mlsanity.com/paper/1908.01768/) (2019)
- [Demystifying TasNet: A Dissecting Approach](https://mlsanity.com/paper/1911.08895/) (2019)
- [End-to-End Multi-Channel Speech Separation](https://mlsanity.com/paper/1905.06286/) (2019)
- [A Multi-Phase Gammatone Filterbank for Speech Separation via TasNet](https://mlsanity.com/paper/1910.11615/) (2019)
- [Interrupted and cascaded permutation invariant training for speech separation](https://mlsanity.com/paper/1910.12706/) (2019)
- [A comprehensive study of speech separation: spectrogram vs waveform separation](https://mlsanity.com/paper/1905.07497/) (2019)
- [Problem-Agnostic Speech Embeddings for Multi-Speaker Text-to-Speech with SampleRNN](https://mlsanity.com/paper/1906.00733/) (2019)
- [Improving Voice Separation by Incorporating End-to-end Speech Recognition](https://mlsanity.com/paper/1911.12928/) (2019)

## Built on by (most cited first, 4 of 4)

- [On loss functions and evaluation metrics for music source separation](https://mlsanity.com/paper/2202.07968/) (2022)
- [Adversarial Permutation Invariant Training for Universal Sound Separation](https://mlsanity.com/paper/2210.12108/) (2022)
- [Individualized Conditioning and Negative Distances for Speaker Separation](https://mlsanity.com/paper/2210.06368/) (2022)
- [Single-Channel Distance-Based Source Separation for Mobile GPU in Outdoor and Indoor Environments](https://mlsanity.com/paper/2501.03045/) (2025)

## Abstract (arXiv)

We study permutation invariant training (PIT), which targets at the permutation ambiguity problem for speaker independent source separation models. We extend two state-of-the-art PIT strategies. First, we look at the two-stage speaker separation and tracking algorithm based on frame level PIT (tPIT) and clustering, which was originally proposed for the STFT domain, and we adapt it to work with waveforms and over a learned latent space. Further, we propose an efficient clustering loss scalable to waveform models. Second, we extend a recently proposed auxiliary speaker-ID loss with a deep feature loss based on "problem agnostic speech features", to reduce the local permutation errors made by the utterance level PIT (uPIT). Our results show that the proposed extensions help reducing permutation ambiguity. However, we also note that the studied STFT-based models are more effective at reducing permutation errors than waveform-based models, a perspective overlooked in recent studies.