mlsanityData to 14 Oct 2026

Diffusion-based Generative Speech Source Separation

arXiv 2210.17327 ↗First posted 31 Oct 2022eess.AS cs.LG cs.SDcited by 41

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose DiffSep, a new single channel source separation method based on score-matching of a stochastic differential equation (SDE). We craft a tailored continuous time diffusion-mixing process starting from the separated sources and converging to a Gaussian distribution centered on their mixture. This formulation lets us apply the machinery of score-based generative modelling. First, we train a neural network to approximate the score function of the marginal probabilities or the diffusion-mixing process. Then, we use it to solve the reverse time SDE that progressively separates the sources starting from their mixture. We propose a modified training strategy to handle model mismatch and source permutation ambiguity. Experiments on the WSJ0 2mix dataset demonstrate the potential of the method. Furthermore, the method is also suitable for speech enhancement and shows performance competitive with prior work on the VoiceBank-DEMAND dataset.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 33 of its 43 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

33 of 43 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  2. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar
  3. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping31 Mar
  4. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  5. 2021A Study on Speech Enhancement Based on Diffusion Probabilistic Model25 Jul
  6. 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior11 Jun
  7. 2021Restoring degraded speech via a modified diffusion model22 Apr
  8. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 25

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  7. 2018SDR - half-baked or well done?cited by 423
  8. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  9. 2021Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Modelscited by 252
  10. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  11. 2020Attention is All You Need in Speech Separationcited by 185
  12. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
13 more
  1. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  2. 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 98
  3. 2020Adversarial score matching and improved sampling for image generationcited by 96
  4. 2020Wavesplit: End-to-End Speech Separation by Speaker Clusteringcited by 75
  5. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  6. 2020Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separationcited by 68
  7. 2022TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separationcited by 48
  8. 2020Source Separation with Deep Generative Priorscited by 36
  9. 2022NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Ratescited by 36
  10. 2021Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Networkcited by 32
  11. 2017Generative Adversarial Source Separationcited by 19
  12. 2021Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamicscited by 19
  13. 2022ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understandingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 8; 2025: 16; 2026: 5.

41 papers.

Sort:

By area:

  1. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  2. 2023Multi-Source Diffusion Models for Simultaneous Music Generation and Separationcited by 63
  3. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  4. 2024Diffusion Models for Audio Restorationcited by 32
  5. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  6. 2025LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancementcited by 19
  7. 2023Separate And Diffuse: Using a Pretrained Diffusion Model for Improving Source Separationcited by 15
  8. 2023Target Speech Extraction with Conditional Diffusion Modelcited by 15
  9. 2024Simultaneous Music Separation and Generation Using Multi-Track Latent Diffusion Modelscited by 13
  10. 2023DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extractioncited by 12
  11. 2023High-Quality Visually-Guided Sound Separation from Diverse Categoriescited by 11
  12. 2023Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Modelcited by 9
29 more
  1. 2024Noise-robust Speech Separation with Fast Generative Correctioncited by 9
  2. 2025SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipelinecited by 9
  3. 2023DDTSE: Discriminative Diffusion Model for Target Speech Extractioncited by 8
  4. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  5. 2025Source Separation by Flow Matchingcited by 8
  6. 2023UnDiff: Unsupervised Voice Restoration with Unconditional Diffusion Modelcited by 7
  7. 2025ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Priorcited by 7
  8. 2024GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Modelcited by 6
  9. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 6
  10. 2025FlowTSE: Target Speaker Extraction with Flow Matchingcited by 6
  11. 2023Conditional Diffusion Model for Target Speaker Extractioncited by 4
  12. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extractioncited by 4
  13. 2025ZeroSep: Separate Anything in Audio with Zero Trainingcited by 4
  14. 2023Diffusion-based Signal Refiner for Speech Enhancement and Separationcited by 3
  15. 2025Bayesian Signal Separation via Plug-and-Play Diffusion-Within-Gibbs Samplingcited by 3
  16. 2025SepALM: Audio Language Models Are Error Correctors for Robust Speech Separationcited by 2
  17. 2025User-guided Generative Source Separationcited by 2
  18. 2025Knowing When to Quit: Probabilistic Early Exits for Speech Separationcited by 2
  19. 2025High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modelingcited by 2
  20. 2025Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Modelcited by 2
  21. 2024Improving Music Source Separation with Diffusion and Consistency Refinementcited by 1
  22. 2025Training-Free Multi-Step Audio Source Separationcited by 1
  23. 2024Biomimetic Frontend for Differentiable Audio Processingcited by 0
  24. 2025GLA-Grad++: An Improved Griffin-Lim Guided Diffusion Model for Speech Synthesiscited by 0
  25. 2026SLM-SS: Speech Language Model for Generative Speech Separationcited by 0
  26. 2026Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matchingcited by 0
  27. 2026SURF: Separation via Unsupervised Remixing Flowcited by 0
  28. 2026MeCo: One-Step MeanFlow-based Corrector for Multi-Channel Speech Separationcited by 0
  29. 2026Iterative Audio Separation with Mixture Consistency via MIMO Model Extensioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Single and Few-step Diffusion for Generative Speech Enhancement
  2. 2023Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement
  3. 2023Unsupervised speech enhancement with diffusion-based generative models
  4. 2024Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement
  5. 2025Audio Generation Through Score-Based Generative Modeling: Design Principles and Implementation
  6. 2024Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
  7. 2025DGMO: Training-Free Audio Source Separation through Diffusion-Guided Mask Optimization
  8. 2026Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation