# Speech Enhancement and Dereverberation with Diffusion-based Generative Models

arXiv: https://arxiv.org/abs/2208.05830 · HTML: https://mlsanity.com/paper/2208.05830/

## Key facts

- Cited by 197 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 55 of its 88 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Conditional Image Generation with Score-Based Diffusion Models](https://mlsanity.com/paper/2111.13606/) (2021)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality](https://mlsanity.com/paper/2202.05830/) (2022)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)

## Built on by (most cited first, 30 of 197)

- [Image Restoration with Mean-Reverting Stochastic Differential Equations](https://mlsanity.com/paper/2301.11699/) (2023)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Soft Diffusion: Score Matching for General Corruptions](https://mlsanity.com/paper/2209.05442/) (2022)
- [An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization](https://mlsanity.com/paper/2404.07771/) (2024)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [Diffusion-based Generative Speech Source Separation](https://mlsanity.com/paper/2210.17327/) (2022)
- [Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation](https://mlsanity.com/paper/2409.14411/) (2024)
- [Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration](https://mlsanity.com/paper/2211.02397/) (2022)
- [Music2Latent: Consistency Autoencoders for Latent Audio Compression](https://mlsanity.com/paper/2408.06500/) (2024)
- [Diffusion Models for Audio Restoration](https://mlsanity.com/paper/2402.09821/) (2024)
- [Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.04080/) (2024)
- [Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement](https://mlsanity.com/paper/2302.14748/) (2023)
- [SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models](https://mlsanity.com/paper/2304.05961/) (2023)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching](https://mlsanity.com/paper/2406.09286/) (2024)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [Blind Audio Bandwidth Extension: A Diffusion-Based Zero-Shot Approach](https://mlsanity.com/paper/2306.01433/) (2023)
- [FlowSE: Flow Matching-based Speech Enhancement](https://mlsanity.com/paper/2508.06840/) (2025)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Diffusion models for audio semantic communication](https://mlsanity.com/paper/2309.07195/) (2023)
- [FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching](https://mlsanity.com/paper/2505.19476/) (2025)
- [Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation](https://mlsanity.com/paper/2511.05516/) (2025)
- [DriftRec: Adapting diffusion models to blind JPEG restoration](https://mlsanity.com/paper/2211.06757/) (2022)
- [Target Speech Extraction with Conditional Diffusion Model](https://mlsanity.com/paper/2308.03987/) (2023)
- [Single and Few-step Diffusion for Generative Speech Enhancement](https://mlsanity.com/paper/2309.09677/) (2023)

## Abstract (arXiv)

In this work, we build upon our previous publication and use diffusion-based generative models for speech enhancement. We present a detailed overview of the diffusion process that is based on a stochastic differential equation and delve into an extensive theoretical examination of its implications. Opposed to usual conditional generation tasks, we do not start the reverse process from pure Gaussian noise but from a mixture of noisy speech and Gaussian noise. This matches our forward process which moves from clean speech to noisy speech by including a drift term. We show that this procedure enables using only 30 diffusion steps to generate high-quality clean speech estimates. By adapting the network architecture, we are able to significantly improve the speech enhancement performance, indicating that the network, rather than the formalism, was the main limitation of our original approach. In an extensive cross-dataset evaluation, we show that the improved method can compete with recent discriminative models and achieves better generalization when evaluating on a different corpus than used for training. We complement the results with an instrumental evaluation using real-world noisy recordings and a listening experiment, in which our proposed method is rated best. Examining different sampler configurations for solving the reverse process allows us to balance the performance and computational speed of the proposed method. Moreover, we show that the proposed method is also suitable for dereverberation and thus not limited to additive background noise removal. Code and audio examples are available online, see https://github.com/sp-uhh/sgmse.