# Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement

arXiv: https://arxiv.org/abs/2406.13471 · HTML: https://mlsanity.com/paper/2406.13471/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 21 of 21)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration](https://mlsanity.com/paper/2011.03706/) (2020)
- [Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modeling](https://mlsanity.com/paper/1903.04567/) (2019)
- [Improving noise robust automatic speech recognition with single-channel time-domain enhancement network](https://mlsanity.com/paper/2003.03998/) (2020)
- [SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separation](https://mlsanity.com/paper/2201.10800/) (2022)
- [ESPnet-SE++: Speech Enhancement for Robust Speech Recognition, Translation, and Understanding](https://mlsanity.com/paper/2207.09514/) (2022)
- [Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions](https://mlsanity.com/paper/2110.14139/) (2021)
- [Speech Signal Improvement Using Causal Generative Diffusion Models](https://mlsanity.com/paper/2303.08674/) (2023)

## Built on by (most cited first, 8 of 8)

- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)
- [SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement](https://mlsanity.com/paper/2509.24708/) (2025)
- [Diffusion Buffer for Online Generative Speech Enhancement](https://mlsanity.com/paper/2510.18744/) (2025)
- [Accelerating Diffusion Sampling via Exploiting Local Transition Coherence](https://mlsanity.com/paper/2503.09675/) (2025)
- [Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement](https://mlsanity.com/paper/2409.09642/) (2024)
- [Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement](https://mlsanity.com/paper/2505.13983/) (2025)
- [RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation](https://mlsanity.com/paper/2608.12099/) (2026)
- [Discriminative Flow Matching: Beyond Time-Conditioning in Generative Restoration via Flow-State Representations](https://mlsanity.com/paper/2609.04525/) (2026)

## Abstract (arXiv)

Diffusion-based generative models (DGMs) have recently attracted attention in speech enhancement research (SE) as previous works showed a remarkable generalization capability. However, DGMs are also computationally intensive, as they usually require many iterations in the reverse diffusion process (RDP), making them impractical for streaming SE systems. In this paper, we propose to use discriminative scores from discriminative models in the first steps of the RDP. These discriminative scores require only one forward pass with the discriminative model for multiple RDP steps, thus greatly reducing computations. This approach also allows for performance improvements. We show that we can trade off between generative and discriminative capabilities as the number of steps with the discriminative score increases. Furthermore, we propose a novel streamable time-domain generative model with an algorithmic latency of 50 ms, which has no significant performance degradation compared to offline models.