# GASS: Generalizing Audio Source Separation with Large-scale Data

arXiv: https://arxiv.org/abs/2310.00140 · HTML: https://mlsanity.com/paper/2310.00140/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 25 of 25)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [Hybrid Transformers for Music Source Separation](https://mlsanity.com/paper/2211.08553/) (2022)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantity](https://mlsanity.com/paper/1909.08494/) (2019)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [The 2018 Signal Separation Evaluation Campaign](https://mlsanity.com/paper/1804.06267/) (2018)
- [Universal Sound Separation](https://mlsanity.com/paper/1905.03330/) (2019)
- [An efficient encoder-decoder architecture with top-down attention for speech separation](https://mlsanity.com/paper/2209.15200/) (2022)
- [CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos](https://mlsanity.com/paper/2212.07065/) (2022)
- [What's All the FUSS About Free Universal Sound Separation Data?](https://mlsanity.com/paper/2011.00803/) (2020)
- [The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks](https://mlsanity.com/paper/2110.09958/) (2021)
- [Improving Universal Sound Separation Using Sound Classification](https://mlsanity.com/paper/1911.07951/) (2019)
- [High Fidelity Speech Enhancement with Band-split RNN](https://mlsanity.com/paper/2212.00406/) (2022)
- [An empirical study of Conv-TasNet](https://mlsanity.com/paper/2002.08688/) (2020)
- [Compute and memory efficient universal sound source separation](https://mlsanity.com/paper/2103.02644/) (2021)
- [Universal Source Separation with Weakly Labelled Data](https://mlsanity.com/paper/2305.07447/) (2023)
- [Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation](https://mlsanity.com/paper/2106.00847/) (2021)
- [Adversarial Permutation Invariant Training for Universal Sound Separation](https://mlsanity.com/paper/2210.12108/) (2022)

## Built on by (most cited first, 6 of 6)

- [Generalized Multi-Source Inference for Text Conditioned Music Diffusion Models](https://mlsanity.com/paper/2403.11706/) (2024)
- [OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup](https://mlsanity.com/paper/2410.21269/) (2024)
- [Learning Source Disentanglement in Neural Audio Codec](https://mlsanity.com/paper/2409.11228/) (2024)
- [DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classification](https://mlsanity.com/paper/2409.12413/) (2024)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [Unleashing the Power of Natural Audio Featuring Multiple Sound Sources](https://mlsanity.com/paper/2504.17782/) (2025)

## Abstract (arXiv)

Universal source separation targets at separating the audio sources of an arbitrary mix, removing the constraint to operate on a specific domain like speech or music. Yet, the potential of universal source separation is limited because most existing works focus on mixes with predominantly sound events, and small training datasets also limit its potential for supervised learning. Here, we study a single general audio source separation (GASS) model trained to separate speech, music, and sound events in a supervised fashion with a large-scale dataset. We assess GASS models on a diverse set of tasks. Our strong in-distribution results show the feasibility of GASS models, and the competitive out-of-distribution performance in sound event and speech separation shows its generalization abilities. Yet, it is challenging for GASS models to generalize for separating out-of-distribution cinematic and music content. We also fine-tune GASS models on each dataset and consistently outperform the ones without pre-training. All fine-tuned models (except the music separation one) obtain state-of-the-art results in their respective benchmarks.