# Sound Source Separation Using Latent Variational Block-Wise Disentanglement

arXiv: https://arxiv.org/abs/2402.06683 · HTML: https://mlsanity.com/paper/2402.06683/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Memformer: A Memory-Augmented Transformer for Sequence Modeling](https://mlsanity.com/paper/2010.06891/) (2020)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [A variance modeling framework based on variational autoencoders for speech enhancement](https://mlsanity.com/paper/1902.01605/) (2019)
- [SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separation](https://mlsanity.com/paper/2201.10800/) (2022)
- [Fisher Efficient Inference of Intractable Models](https://mlsanity.com/paper/1805.07454/) (2018)
- [A Statistically Principled and Computationally Efficient Approach to Speech Enhancement using Variational Autoencoders](https://mlsanity.com/paper/1905.01209/) (2019)
- [Fast MVAE: Joint separation and classification of mixed sources based on multichannel variational autoencoder with auxiliary classifier](https://mlsanity.com/paper/1812.06391/) (2018)
- [Integration of variational autoencoder and spatial clustering for adaptive multi-channel neural speech separation](https://mlsanity.com/paper/2011.11984/) (2020)

## Built on by (most cited first, 2 of 2)

- [Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement](https://mlsanity.com/paper/2409.09642/) (2024)
- [A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI](https://mlsanity.com/paper/2603.13379/) (2026)

## Abstract (arXiv)

While neural network approaches have made significant strides in resolving classical signal processing problems, it is often the case that hybrid approaches that draw insight from both signal processing and neural networks produce more complete solutions. In this paper, we present a hybrid classical digital signal processing/deep neural network (DSP/DNN) approach to source separation (SS) highlighting the theoretical link between variational autoencoder and classical approaches to SS. We propose a system that transforms the single channel under-determined SS task to an equivalent multichannel over-determined SS problem in a properly designed latent space. The separation task in the latent space is treated as finding a variational block-wise disentangled representation of the mixture. We show empirically, that the design choices and the variational formulation of the task at hand motivated by the classical signal processing theoretical results lead to robustness to unseen out-of-distribution data and reduction of the overfitting risk. To address the resulting permutation issue we explicitly incorporate a novel differentiable permutation loss function and augment the model with a memory mechanism to keep track of the statistics of the individual sources.