# Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain

arXiv: https://arxiv.org/abs/2203.17004 · HTML: https://mlsanity.com/paper/2203.17004/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [A variance modeling framework based on variational autoencoders for speech enhancement](https://mlsanity.com/paper/1902.01605/) (2019)
- [Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder](https://mlsanity.com/paper/2102.08706/) (2021)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modeling](https://mlsanity.com/paper/1903.04567/) (2019)
- [A consolidated view of loss functions for supervised deep learning-based speech enhancement](https://mlsanity.com/paper/2009.12286/) (2020)
- [Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2105.08970/) (2021)

## Built on by (most cited first, 30 of 102)

- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Image Restoration with Mean-Reverting Stochastic Differential Equations](https://mlsanity.com/paper/2301.11699/) (2023)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization](https://mlsanity.com/paper/2404.07771/) (2024)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis](https://mlsanity.com/paper/2205.14807/) (2022)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration](https://mlsanity.com/paper/2211.02397/) (2022)
- [Diffusion Models for Audio Restoration](https://mlsanity.com/paper/2402.09821/) (2024)
- [Cold Diffusion for Speech Enhancement](https://mlsanity.com/paper/2211.02527/) (2022)
- [Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.04080/) (2024)
- [Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement](https://mlsanity.com/paper/2302.14748/) (2023)
- [Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation](https://mlsanity.com/paper/2311.04693/) (2023)
- [Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution](https://mlsanity.com/paper/2210.15793/) (2022)
- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching](https://mlsanity.com/paper/2505.19476/) (2025)
- [DriftRec: Adapting diffusion models to blind JPEG restoration](https://mlsanity.com/paper/2211.06757/) (2022)
- [Target Speech Extraction with Conditional Diffusion Model](https://mlsanity.com/paper/2308.03987/) (2023)
- [Single and Few-step Diffusion for Generative Speech Enhancement](https://mlsanity.com/paper/2309.09677/) (2023)
- [FlowDec: A flow-based full-band general audio codec with high perceptual quality](https://mlsanity.com/paper/2503.01485/) (2025)
- [Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement](https://mlsanity.com/paper/2210.17287/) (2022)
- [Audio-Visual Speech Enhancement with Score-Based Generative Models](https://mlsanity.com/paper/2306.01432/) (2023)
- [Unsupervised speech enhancement with diffusion-based generative models](https://mlsanity.com/paper/2309.10450/) (2023)
- [Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation with Diffusion Models](https://mlsanity.com/paper/2408.07472/) (2024)
- [Investigating the Design Space of Diffusion Models for Speech Enhancement](https://mlsanity.com/paper/2312.04370/) (2023)
- [On the Asymptotic Mean Square Error Optimality of Diffusion Models](https://mlsanity.com/paper/2403.02957/) (2024)
- [Diffusion Posterior Sampling for Informed Single-Channel Dereverberation](https://mlsanity.com/paper/2306.12286/) (2023)

## Abstract (arXiv)

Score-based generative models (SGMs) have recently shown impressive results for difficult generative tasks such as the unconditional and conditional generation of natural images and audio signals. In this work, we extend these models to the complex short-time Fourier transform (STFT) domain, proposing a novel training task for speech enhancement using a complex-valued deep neural network. We derive this training task within the formalism of stochastic differential equations (SDEs), thereby enabling the use of predictor-corrector samplers. We provide alternative formulations inspired by previous publications on using generative diffusion models for speech enhancement, avoiding the need for any prior assumptions on the noise distribution and making the training task purely generative which, as we show, results in improved enhancement performance.