# Unsupervised Music Source Separation Using Differentiable Parametric Source Models

arXiv: https://arxiv.org/abs/2201.09592 · HTML: https://mlsanity.com/paper/2201.09592/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 73 references that are in the map.
- First posted on arXiv 24 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [The 2018 Signal Separation Evaluation Campaign](https://mlsanity.com/paper/1804.06267/) (2018)
- [Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separations](https://mlsanity.com/paper/1907.01277/) (2019)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [Neural source-filter-based waveform model for statistical parametric speech synthesis](https://mlsanity.com/paper/1810.11946/) (2018)
- [Unsupervised Audio Source Separation using Generative Priors](https://mlsanity.com/paper/2005.13769/) (2020)
- [Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization](https://mlsanity.com/paper/1811.06713/) (2018)
- [Unsupervised training of a deep clustering model for multichannel blind source separation](https://mlsanity.com/paper/1904.01340/) (2019)
- [Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information](https://mlsanity.com/paper/1811.01531/) (2018)
- [Deep Learning Based Source Separation Applied To Choir Ensembles](https://mlsanity.com/paper/2008.07645/) (2020)
- [Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision](https://mlsanity.com/paper/2010.11904/) (2020)
- [Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation](https://mlsanity.com/paper/2106.00847/) (2021)
- [Automatic Lyrics Transcription using Dilated Convolutional Neural Networks with Self-Attention](https://mlsanity.com/paper/2007.06486/) (2020)
- [Multiple F0 Estimation in Vocal Ensembles using Convolutional Neural Networks](https://mlsanity.com/paper/2009.04172/) (2020)

## Built on by (most cited first, 4 of 4)

- [Zero-Shot Duet Singing Voices Separation with Diffusion Models](https://mlsanity.com/paper/2311.07345/) (2023)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis](https://mlsanity.com/paper/2505.03337/) (2025)
- [Reconstructing the Vocal Tract with Differentiable Acoustic Simulation](https://mlsanity.com/paper/2609.36737/) (2026)

## Abstract (arXiv)

Supervised deep learning approaches to underdetermined audio source separation achieve state-of-the-art performance but require a dataset of mixtures along with their corresponding isolated source signals. Such datasets can be extremely costly to obtain for musical mixtures. This raises a need for unsupervised methods. We propose a novel unsupervised model-based deep learning approach to musical source separation. Each source is modelled with a differentiable parametric source-filter model. A neural network is trained to reconstruct the observed mixture as a sum of the sources by estimating the source models' parameters given their fundamental frequencies. At test time, soft masks are obtained from the synthesized source signals. The experimental evaluation on a vocal ensemble separation task shows that the proposed method outperforms learning-free methods based on nonnegative matrix factorization and a supervised deep learning baseline. Integrating domain knowledge in the form of source models into a data-driven method leads to high data efficiency: the proposed approach achieves good separation quality even when trained on less than three minutes of audio. This work makes powerful deep learning based separation usable in scenarios where training data with ground truth is expensive or nonexistent.