# Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervision

arXiv: https://arxiv.org/abs/1911.02182 · HTML: https://mlsanity.com/paper/1911.02182/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 13 of 13)

- [Visual Scene Graphs for Audio Source Separation](https://mlsanity.com/paper/2109.11955/) (2021)
- [AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation](https://mlsanity.com/paper/2207.10141/) (2022)
- [Weakly-Supervised Audio-Visual Segmentation](https://mlsanity.com/paper/2311.15080/) (2023)
- [Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data](https://mlsanity.com/paper/2105.04727/) (2021)
- [MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training](https://mlsanity.com/paper/2202.03875/) (2022)
- [Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention](https://mlsanity.com/paper/2106.09669/) (2021)
- [Weakly-supervised Audio Separation via Bi-modal Semantic Similarity](https://mlsanity.com/paper/2404.01740/) (2024)
- [Semantic Grouping Network for Audio Source Separation](https://mlsanity.com/paper/2407.03736/) (2024)
- [OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation](https://mlsanity.com/paper/2409.19270/) (2024)
- [MACS: Multi-source Audio-to-image Generation with Contextual Significance and Semantic Alignment](https://mlsanity.com/paper/2503.10287/) (2025)
- [Unsupervised Source Separation By Steering Pretrained Music Models](https://mlsanity.com/paper/2110.13071/) (2021)
- [Sampling-Frequency-Independent Universal Sound Separation](https://mlsanity.com/paper/2309.12581/) (2023)
- [Text-Queried Audio Source Separation via Hierarchical Modeling](https://mlsanity.com/paper/2505.21025/) (2025)

## Abstract (arXiv)

While there has been much recent progress using deep learning techniques to separate speech and music audio signals, these systems typically require large collections of isolated sources during the training process. When extending audio source separation algorithms to more general domains such as environmental monitoring, it may not be possible to obtain isolated signals for training. Here, we propose objective functions and network architectures that enable training a source separation system with weak labels. In this scenario, weak labels are defined in contrast with strong time-frequency (TF) labels such as those obtained from isolated sources, and refer either to frame-level weak labels where one only has access to the time periods when different sources are active in an audio mixture, or to clip-level weak labels that only indicate the presence or absence of sounds in an entire audio clip. We train a separator that estimates a TF mask for each type of sound event, using a sound event classifier as an assessor of the separator's performance to bridge the gap between the TF-level separation and the ground truth weak labels only available at the frame or clip level. Our objective function requires the classifier applied to a separated source to assign high probability to the class corresponding to that source and low probability to all other classes. The objective function also enforces that the separated sources sum up to the mixture. We benchmark the performance of our algorithm using synthetic mixtures of overlapping events created from a database of sounds recorded in urban environments. Compared to training a network using isolated sources, our model achieves somewhat lower but still significant SI-SDR improvement, even in scenarios with significant sound event overlap.