mlsanityData to 7 Oct 2026

Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

arXiv 2203.17004 ↗First posted 31 Mar 2022eess.AS cs.LG cs.SDcited by 102

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Score-based generative models (SGMs) have recently shown impressive results for difficult generative tasks such as the unconditional and conditional generation of natural images and audio signals. In this work, we extend these models to the complex short-time Fourier transform (STFT) domain, proposing a novel training task for speech enhancement using a complex-valued deep neural network. We derive this training task within the formalism of stochastic differential equations (SDEs), thereby enabling the use of predictor-corrector samplers. We provide alternative formulations inspired by previous publications on using generative diffusion models for speech enhancement, avoiding the need for any prior assumptions on the noise distribution and making the training task purely generative which, as we show, results in improved enhancement performance.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 21 of its 47 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

21 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  2. 2021A Study on Speech Enhancement Based on Diffusion Probabilistic Model25 Jul
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  7. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  8. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  9. 2018SDR - half-baked or well done?cited by 420
  10. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 165
  11. 2019Phase-aware Speech Enhancement with Deep Complex U-Netcited by 73
  12. 2017Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorizationcited by 33
6 more
  1. 2019A variance modeling framework based on variational autoencoders for speech enhancementcited by 33
  2. 2021Variational Autoencoder for Speech Enhancement with a Noise-Aware Encodercited by 28
  3. 2021Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoderscited by 28
  4. 2019Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modelingcited by 21
  5. 2020A consolidated view of loss functions for supervised deep learning-based speech enhancementcited by 19
  6. 2021Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancementcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 23; 2024: 17; 2025: 37; 2026: 14.

102 papers.

Sort:

By area:

  1. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 195
  2. 2023Image Restoration with Mean-Reverting Stochastic Differential Equationscited by 182
  3. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberationcited by 105
  4. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  5. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  6. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  7. 2024A Survey on Diffusion Models for Time Series and Spatio-Temporal Datacited by 66
  8. 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesiscited by 44
  9. 2022Diffusion-based Generative Speech Source Separationcited by 41
  10. 2022Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restorationcited by 34
  11. 2024Diffusion Models for Audio Restorationcited by 32
  12. 2022Cold Diffusion for Speech Enhancementcited by 31
38 more
  1. 2023Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancementcited by 30
  2. 2023Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptationcited by 30
  3. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 29
  4. 2022Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolutioncited by 20
  5. 2025LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancementcited by 19
  6. 2023The Ethical Implications of Generative Audio Models: A Systematic Literature Reviewcited by 18
  7. 2025FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matchingcited by 17
  8. 2022DriftRec: Adapting diffusion models to blind JPEG restorationcited by 15
  9. 2023Target Speech Extraction with Conditional Diffusion Modelcited by 15
  10. 2023Single and Few-step Diffusion for Generative Speech Enhancementcited by 15
  11. 2025FlowDec: A flow-based full-band general audio codec with high perceptual qualitycited by 15
  12. 2022Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancementcited by 14
  13. 2023Audio-Visual Speech Enhancement with Score-Based Generative Modelscited by 12
  14. 2023Unsupervised speech enhancement with diffusion-based generative modelscited by 12
  15. 2024Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation with Diffusion Modelscited by 11
  16. 2023Investigating the Design Space of Diffusion Models for Speech Enhancementcited by 10
  17. 2024On the Asymptotic Mean Square Error Optimality of Diffusion Modelscited by 10
  18. 2023Diffusion Posterior Sampling for Informed Single-Channel Dereverberationcited by 9
  19. 2023Noise-aware Speech Enhancement using Diffusion Probabilistic Modelcited by 9
  20. 2022DiffPhase: Generative Diffusion-based STFT Phase Retrievalcited by 8
  21. 2023DDTSE: Discriminative Diffusion Model for Target Speech Extractioncited by 8
  22. 2023Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions Using a Heun-Based Samplercited by 8
  23. 2024Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancementcited by 8
  24. 2025Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representationcited by 8
  25. 2023SE-Bridge: Speech Enhancement with Consistent Brownian Bridgecited by 7
  26. 2024FINALLY: fast and universal speech enhancement with studio-like qualitycited by 7
  27. 2025Speech Enhancement Using Continuous Embeddings of Neural Audio Codeccited by 7
  28. 2025Bilingual Dual-Head Deep Model for Parkinson's Disease Detection from Speechcited by 7
  29. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  30. 2024Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Modelcited by 6
  31. 2024Thunder : Unified Regression-Diffusion Speech Enhancement with a Single Reverse Step using Brownian Bridgecited by 6
  32. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  33. 2025Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latencycited by 6
  34. 2025MeanFlowSE: one-step generative speech enhancement via conditional mean flowcited by 6
  35. 2023Variance-Preserving-Based Interpolation Diffusion Models for Speech Enhancementcited by 5
  36. 2024An Analysis of the Variance of Diffusion-based Speech Enhancementcited by 5
  37. 2024A Variance-Preserving Interpolation Approach for Diffusion Models with Applications to Single Channel Speech Enhancement and Recognitioncited by 5
  38. 2024Stability and Generalizability in SDE Diffusion Models with Measure-Preserving Dynamicscited by 5

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating Training Objectives for Generative Speech Enhancement
  2. 2023Diffusion-based speech enhancement with a weighted generative-supervised learning loss
  3. 2022SRTNet: Time Domain Speech Enhancement Via Stochastic Refinement
  4. 2024Diffusion-based Speech Enhancement with Schr"odinger Bridge and Symmetric Noise Schedule
  5. 2026Rethinking Language Model-Based Generative Speech Enhancement in the Latent Space of a Neural Audio Codec
  6. 2024Complex Image-Generative Diffusion Transformer for Audio Denoising
  7. 2023NADiffuSE: Noise-aware Diffusion-based Model for Speech Enhancement
  8. 2023uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models