mlsanityData to 7 Oct 2026

Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement

arXiv 2409.09642 ↗First posted 15 Sep 2024eess.AS cs.LG cs.SDcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Diffusion-based generative models have recently achieved remarkable results in speech and vocal enhancement due to their ability to model complex speech data distributions. While these models generalize well to unseen acoustic environments, they may not achieve the same level of fidelity as the discriminative models specifically trained to enhance particular acoustic conditions. In this paper, we propose Ex-Diff, a novel score-based diffusion model that integrates the latent representations produced by a discriminative model to improve speech and vocal enhancement, which combines the strengths of both generative and discriminative models. Experimental results on the widely used MUSDB dataset show relative improvements of 3.7% in SI-SDR and 10.0% in SI-SIR compared to the baseline diffusion model for speech and vocal enhancement tasks, respectively. Additionally, case studies are provided to further illustrate and analyze the complementary nature of generative and discriminative models in this context.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 29 of its 46 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

29 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Diffusion-based Speech Enhancement with Schr"odinger Bridge and Symmetric Noise Schedule8 Sep
  2. 2024Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement19 Jun
  3. 2022Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration4 Nov
  4. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  5. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2015Variational Inference with Normalizing Flowscited by 2,161
  8. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  9. 2018SDR - half-baked or well done?cited by 420
  10. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  11. 2019Music Source Separation in the Waveform Domaincited by 98
  12. 2024An Investigation of Incorporating Mamba for Speech Enhancementcited by 61
12 more
  1. 2019Demucs: Deep Extractor for Music Sources with extra unlabeled data remixedcited by 53
  2. 2020Sudo rm -rf: Efficient Networks for Universal Audio Source Separationcited by 50
  3. 2021Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separationcited by 48
  4. 2020Speech Enhancement using Self-Adaptation and Multi-Head Self-Attentioncited by 28
  5. 2021Variational Autoencoder for Speech Enhancement with a Noise-Aware Encodercited by 28
  6. 2024Schr"odinger Bridge for Generative Speech Enhancementcited by 27
  7. 2023Universal Source Separation with Weakly Labelled Datacited by 15
  8. 2023Toward Universal Speech Enhancement for Diverse Input Conditionscited by 10
  9. 2021Self-Supervised Learning from Automatically Separated Sound Scenescited by 6
  10. 2022Latent Iterative Refinement for Modular Source Separationcited by 6
  11. 2023A Framework for Unified Real-time Personalized and Non-Personalized Speech Enhancementcited by 2
  12. 2024Sound Source Separation Using Latent Variational Block-Wise Disentanglementcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Single and Few-step Diffusion for Generative Speech Enhancement
  2. 2023Diffusion-based speech enhancement with a weighted generative-supervised learning loss
  3. 2023Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions Using a Heun-Based Sampler
  4. 2023Unsupervised speech enhancement with diffusion-based generative models
  5. 2025ProSE: Diffusion Priors for Speech Enhancement
  6. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  7. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
  8. 2025Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement