# Adversarial Permutation Invariant Training for Universal Sound Separation

arXiv: https://arxiv.org/abs/2210.12108 · HTML: https://mlsanity.com/paper/2210.12108/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)
- [Upsampling artifacts in neural audio synthesis](https://mlsanity.com/paper/2010.14356/) (2020)
- [What's All the FUSS About Free Universal Sound Separation Data?](https://mlsanity.com/paper/2011.00803/) (2020)
- [Generative Adversarial Source Separation](https://mlsanity.com/paper/1710.10779/) (2017)
- [End-to-end music source separation: is it possible in the waveform domain?](https://mlsanity.com/paper/1810.12187/) (2018)
- [Unsupervised Audio Source Separation using Generative Priors](https://mlsanity.com/paper/2005.13769/) (2020)
- [On loss functions and evaluation metrics for music source separation](https://mlsanity.com/paper/2202.07968/) (2022)
- [An empirical study of Conv-TasNet](https://mlsanity.com/paper/2002.08688/) (2020)
- [Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction](https://mlsanity.com/paper/1711.00048/) (2017)
- [Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation](https://mlsanity.com/paper/2106.00847/) (2021)
- [On permutation invariant training for speech source separation](https://mlsanity.com/paper/2102.04945/) (2021)

## Built on by (most cited first, 8 of 8)

- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Fast Text-to-Audio Generation with Adversarial Post-Training](https://mlsanity.com/paper/2505.08175/) (2025)
- [Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models](https://mlsanity.com/paper/2409.12346/) (2024)
- [Latent Autoregressive Source Separation](https://mlsanity.com/paper/2301.08562/) (2023)
- [GASS: Generalizing Audio Source Separation with Large-scale Data](https://mlsanity.com/paper/2310.00140/) (2023)
- [Improving Music Source Separation with Diffusion and Consistency Refinement](https://mlsanity.com/paper/2412.06965/) (2024)
- [MARS-Sep: Multimodal-Aligned Reinforced Sound Separation](https://mlsanity.com/paper/2510.10509/) (2025)
- [Iterative Audio Separation with Mixture Consistency via MIMO Model Extension](https://mlsanity.com/paper/2609.07226/) (2026)

## Abstract (arXiv)

Universal sound separation consists of separating mixes with arbitrary sounds of different types, and permutation invariant training (PIT) is used to train source agnostic models that do so. In this work, we complement PIT with adversarial losses but find it challenging with the standard formulation used in speech source separation. We overcome this challenge with a novel I-replacement context-based adversarial loss, and by training with multiple discriminators. Our experiments show that by simply improving the loss (keeping the same model and dataset) we obtain a non-negligible improvement of 1.4 dB SI-SNRi in the reverberant FUSS dataset. We also find adversarial PIT to be effective at reducing spectral holes, ubiquitous in mask-based separation models, which highlights the potential relevance of adversarial losses for source separation.