# Universal Source Separation with Weakly Labelled Data

arXiv: https://arxiv.org/abs/2305.07447 · HTML: https://mlsanity.com/paper/2305.07447/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2023.

## Built on by (most cited first, 15 of 15)

- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Learning Musical Representations for Music Performance Question Answering](https://mlsanity.com/paper/2502.06710/) (2025)
- [A Stem-Agnostic Single-Decoder System for Music Source Separation Beyond Four Stems](https://mlsanity.com/paper/2406.18747/) (2024)
- [GASS: Generalizing Audio Source Separation with Large-scale Data](https://mlsanity.com/paper/2310.00140/) (2023)
- [MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction](https://mlsanity.com/paper/2505.23305/) (2025)
- [DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization](https://mlsanity.com/paper/2506.02858/) (2025)
- [Universal Sound Separation with Self-Supervised Audio Masked Autoencoder](https://mlsanity.com/paper/2407.11745/) (2024)
- [SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separation](https://mlsanity.com/paper/2601.19702/) (2026)
- [FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation](https://mlsanity.com/paper/2608.22111/) (2026)
- [Sampling-Frequency-Independent Universal Sound Separation](https://mlsanity.com/paper/2309.12581/) (2023)
- [Performance Improvement of Language-Queried Audio Source Separation Based on Caption Augmentation From Large Language Models for DCASE Challenge 2024 Task 9](https://mlsanity.com/paper/2406.11248/) (2024)
- [Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement](https://mlsanity.com/paper/2409.09642/) (2024)
- [Text-Queried Audio Source Separation via Hierarchical Modeling](https://mlsanity.com/paper/2505.21025/) (2025)
- [A Conditioned UNet for Music Source Separation](https://mlsanity.com/paper/2512.15532/) (2025)

## Abstract (arXiv)

Universal source separation (USS) is a fundamental research task for computational auditory scene analysis, which aims to separate mono recordings into individual source tracks. There are three potential challenges awaiting the solution to the audio source separation task. First, previous audio source separation systems mainly focus on separating one or a limited number of specific sources. There is a lack of research on building a unified system that can separate arbitrary sources via a single model. Second, most previous systems require clean source data to train a separator, while clean source data are scarce. Third, there is a lack of USS system that can automatically detect and separate active sound classes in a hierarchical level. To use large-scale weakly labeled/unlabeled audio data for audio source separation, we propose a universal audio source separation framework containing: 1) an audio tagging model trained on weakly labeled data as a query net; and 2) a conditional source separation model that takes query net outputs as conditions to separate arbitrary sound sources. We investigate various query nets, source separation models, and training strategies and propose a hierarchical USS strategy to automatically detect and separate sound classes from the AudioSet ontology. By solely leveraging the weakly labelled AudioSet, our USS system is successful in separating a wide variety of sound classes, including sound event separation, music source separation, and speech enhancement. The USS system achieves an average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over 527 sound classes of AudioSet; 10.57 dB on the DCASE 2018 Task 2 dataset; 8.12 dB on the MUSDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of 9.00 dB on the voicebank-demand dataset. We release the source code at https://github.com/bytedance/uss