# Single and Few-step Diffusion for Generative Speech Enhancement

arXiv: https://arxiv.org/abs/2309.09677 · HTML: https://mlsanity.com/paper/2309.09677/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 15 of 15)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration](https://mlsanity.com/paper/2211.02397/) (2022)
- [Cold Diffusion for Speech Enhancement](https://mlsanity.com/paper/2211.02527/) (2022)
- [Reducing the Prior Mismatch of Stochastic Differential Equations for Diffusion-based Speech Enhancement](https://mlsanity.com/paper/2302.14748/) (2023)

## Built on by (most cited first, 15 of 15)

- [Diffusion Models for Audio Restoration](https://mlsanity.com/paper/2402.09821/) (2024)
- [FlowSE: Flow Matching-based Speech Enhancement](https://mlsanity.com/paper/2508.06840/) (2025)
- [DDTSE: Discriminative Diffusion Model for Target Speech Extraction](https://mlsanity.com/paper/2309.13874/) (2023)
- [Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model](https://mlsanity.com/paper/2402.10642/) (2024)
- [MeanFlowSE: one-step generative speech enhancement via conditional mean flow](https://mlsanity.com/paper/2509.14858/) (2025)
- [Robust One-step Speech Enhancement via Consistency Distillation](https://mlsanity.com/paper/2507.05688/) (2025)
- [Towards Real-Time Generative Speech Restoration with Flow-Matching](https://mlsanity.com/paper/2510.16997/) (2025)
- [Shortcut Flow Matching for Speech Enhancement: Step-Invariant flows via single stage training](https://mlsanity.com/paper/2509.21522/) (2025)
- [Diffusion Buffer for Online Generative Speech Enhancement](https://mlsanity.com/paper/2510.18744/) (2025)
- [Rethinking Flow and Diffusion Bridge Models for Speech Enhancement](https://mlsanity.com/paper/2602.18355/) (2026)
- [Robustness of Speech Separation Models for Similar-pitch Speakers](https://mlsanity.com/paper/2407.15749/) (2024)
- [Speech Enhancement based on cascaded two flows](https://mlsanity.com/paper/2508.06842/) (2025)
- [DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformers](https://mlsanity.com/paper/2603.21608/) (2026)
- [Beyond U-Net: A Latent-Representation-Aligned Skip-Free Backbone for Flow-Matching Speech Enhancement](https://mlsanity.com/paper/2606.24745/) (2026)
- [Corrective Forcing: Unified Post-Training for Diffusions and Flows in Generative Speech Enhancement](https://mlsanity.com/paper/2609.24651/) (2026)

## Abstract (arXiv)

Diffusion models have shown promising results in speech enhancement, using a task-adapted diffusion process for the conditional generation of clean speech given a noisy mixture. However, at test time, the neural network used for score estimation is called multiple times to solve the iterative reverse process. This results in a slow inference process and causes discretization errors that accumulate over the sampling trajectory. In this paper, we address these limitations through a two-stage training approach. In the first stage, we train the diffusion model the usual way using the generative denoising score matching loss. In the second stage, we compute the enhanced signal by solving the reverse process and compare the resulting estimate to the clean speech target using a predictive loss. We show that using this second training stage enables achieving the same performance as the baseline model using only 5 function evaluations instead of 60 function evaluations. While the performance of usual generative diffusion algorithms drops dramatically when lowering the number of function evaluations (NFEs) to obtain single-step diffusion, we show that our proposed method keeps a steady performance and therefore largely outperforms the diffusion baseline in this setting and also generalizes better than its predictive counterpart.