# Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separation

arXiv: https://arxiv.org/abs/2301.10752 · HTML: https://mlsanity.com/paper/2301.10752/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 25 Jan 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation](https://mlsanity.com/paper/1602.03253/) (2016)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [TasNet: time-domain audio separation network for real-time, single-channel speech separation](https://mlsanity.com/paper/1711.00541/) (2017)
- [Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networks](https://mlsanity.com/paper/1703.06284/) (2017)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis](https://mlsanity.com/paper/2106.09660/) (2021)
- [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [Source Separation with Deep Generative Priors](https://mlsanity.com/paper/2002.07942/) (2020)
- [SepIt: Approaching a Single Channel Speech Separation Bound](https://mlsanity.com/paper/2205.11801/) (2022)
- [FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks](https://mlsanity.com/paper/1902.04891/) (2019)
- [Many-Speakers Single Channel Speech Separation with Optimal Permutation Training](https://mlsanity.com/paper/2104.08955/) (2021)

## Built on by (most cited first, 15 of 15)

- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Models](https://mlsanity.com/paper/2409.12346/) (2024)
- [Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model](https://mlsanity.com/paper/2310.19581/) (2023)
- [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025)
- [Score-based Source Separation with Applications to Digital Communication Signals](https://mlsanity.com/paper/2306.14411/) (2023)
- [Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2404.12725/) (2024)
- [AudioSlots: A slot-centric generative model for audio separation](https://mlsanity.com/paper/2305.05591/) (2023)
- [DiffSED: Sound Event Detection with Denoising Diffusion](https://mlsanity.com/paper/2308.07293/) (2023)
- [Diffusion-based Signal Refiner for Speech Enhancement and Separation](https://mlsanity.com/paper/2305.05857/) (2023)
- [User-guided Generative Source Separation](https://mlsanity.com/paper/2507.01339/) (2025)
- [Knowing When to Quit: Probabilistic Early Exits for Speech Separation](https://mlsanity.com/paper/2507.09768/) (2025)
- [Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation](https://mlsanity.com/paper/2506.08457/) (2025)
- [AMDM-SE: Attention-based Multichannel Diffusion Model for Speech Enhancement](https://mlsanity.com/paper/2601.13140/) (2026)
- [MeCo: One-Step MeanFlow-based Corrector for Multi-Channel Speech Separation](https://mlsanity.com/paper/2606.09677/) (2026)
- [VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation](https://mlsanity.com/paper/2610.09334/) (2026)

## Abstract (arXiv)

The problem of speech separation, also known as the cocktail party problem, refers to the task of isolating a single speech signal from a mixture of speech signals. Previous work on source separation derived an upper bound for the source separation task in the domain of human speech. This bound is derived for deterministic models. Recent advancements in generative models challenge this bound. We show how the upper bound can be generalized to the case of random generative models. Applying a diffusion model Vocoder that was pretrained to model single-speaker voices on the output of a deterministic separation model leads to state-of-the-art separation results. It is shown that this requires one to combine the output of the separation model with that of the diffusion model. In our method, a linear combination is performed, in the frequency domain, using weights that are inferred by a learned model. We show state-of-the-art results on 2, 3, 5, 10, and 20 speakers on multiple benchmarks. In particular, for two speakers, our method is able to surpass what was previously considered the upper performance bound.