# Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation

arXiv: https://arxiv.org/abs/2106.00847 · HTML: https://mlsanity.com/paper/2106.00847/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2021.

## Built on by (most cited first, 9 of 9)

- [CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos](https://mlsanity.com/paper/2212.07065/) (2022)
- [Adversarial Permutation Invariant Training for Universal Sound Separation](https://mlsanity.com/paper/2210.12108/) (2022)
- [TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition](https://mlsanity.com/paper/2308.10415/) (2023)
- [MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training](https://mlsanity.com/paper/2202.03875/) (2022)
- [GASS: Generalizing Audio Source Separation with Large-scale Data](https://mlsanity.com/paper/2310.00140/) (2023)
- [Weakly-supervised Audio Separation via Bi-modal Semantic Similarity](https://mlsanity.com/paper/2404.01740/) (2024)
- [Unsupervised Music Source Separation Using Differentiable Parametric Source Models](https://mlsanity.com/paper/2201.09592/) (2022)
- [OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation](https://mlsanity.com/paper/2409.19270/) (2024)
- [Text-Queried Audio Source Separation via Hierarchical Modeling](https://mlsanity.com/paper/2505.21025/) (2025)

## Abstract (arXiv)

Supervised neural network training has led to significant progress on single-channel sound separation. This approach relies on ground truth isolated sources, which precludes scaling to widely available mixture data and limits progress on open-domain tasks. The recent mixture invariant training (MixIT) method enables training on in-the-wild data; however, it suffers from two outstanding problems. First, it produces models which tend to over-separate, producing more output sources than are present in the input. Second, the exponential computational complexity of the MixIT loss limits the number of feasible output sources. In this paper we address both issues. To combat over-separation we introduce new losses: sparsity losses that favor fewer output sources and a covariance loss that discourages correlated outputs. We also experiment with a semantic classification loss by predicting weak class labels for each mixture. To handle larger numbers of sources, we introduce an efficient approximation using a fast least-squares solution, projected onto the MixIT constraint set. Our experiments show that the proposed losses curtail over-separation and improve overall performance. The best performance is achieved using larger numbers of output sources, enabled by our efficient MixIT loss, combined with sparsity losses to prevent over-separation. On the FUSS test set, we achieve over 13 dB in multi-source SI-SNR improvement, while boosting single-source reconstruction SI-SNR by over 17 dB.