# The Effect of Training Dataset Size on Discriminative and Diffusion-Based Speech Enhancement Systems

arXiv: https://arxiv.org/abs/2406.06160 · HTML: https://mlsanity.com/paper/2406.06160/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation](https://mlsanity.com/paper/2303.00848/) (2023)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models](https://mlsanity.com/paper/2304.12526/) (2023)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration](https://mlsanity.com/paper/2211.02397/) (2022)
- [Cold Diffusion for Speech Enhancement](https://mlsanity.com/paper/2211.02527/) (2022)
- [MANNER: Multi-view Attention Network for Noise Erasure](https://mlsanity.com/paper/2203.02181/) (2022)
- [SRTNet: Time Domain Speech Enhancement Via Stochastic Refinement](https://mlsanity.com/paper/2210.16805/) (2022)
- [Unsupervised speech enhancement with diffusion-based generative models](https://mlsanity.com/paper/2309.10450/) (2023)
- [Metric-oriented Speech Enhancement using Diffusion Probabilistic Model](https://mlsanity.com/paper/2302.11989/) (2023)
- [Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders](https://mlsanity.com/paper/2305.10734/) (2023)
- [Investigating the Design Space of Diffusion Models for Speech Enhancement](https://mlsanity.com/paper/2312.04370/) (2023)
- [Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions Using a Heun-Based Sampler](https://mlsanity.com/paper/2312.02683/) (2023)
- [Diffusion-based speech enhancement with a weighted generative-supervised learning loss](https://mlsanity.com/paper/2309.10457/) (2023)
- [SNR-Based Features and Diverse Training Data for Robust DNN-Based Speech Enhancement](https://mlsanity.com/paper/2004.03512/) (2020)

## Built on by (most cited first, 3 of 3)

- [Diffusion-based Unsupervised Audio-visual Speech Enhancement](https://mlsanity.com/paper/2410.05301/) (2024)
- [Performance and Efficiency of Climate In-Situ Data Reconstruction: Why Optimized IDW Outperforms kriging and Implicit Neural Representation](https://mlsanity.com/paper/2512.11832/) (2025)
- [Absorbing Discrete Diffusion for Speech Enhancement](https://mlsanity.com/paper/2602.22417/) (2026)

## Abstract (arXiv)

The performance of deep neural network-based speech enhancement systems typically increases with the training dataset size. However, studies that investigated the effect of training dataset size on speech enhancement performance did not consider recent approaches, such as diffusion-based generative models. Diffusion models are typically trained with massive datasets for image generation tasks, but whether this is also required for speech enhancement is unknown. Moreover, studies that investigated the effect of training dataset size did not control for the data diversity. It is thus unclear whether the performance improvement was due to the increased dataset size or diversity. Therefore, we systematically investigate the effect of training dataset size on the performance of popular state-of-the-art discriminative and diffusion-based speech enhancement systems in matched conditions. We control for the data diversity by using a fixed set of speech utterances, noise segments and binaural room impulse responses to generate datasets of different sizes. We find that the diffusion-based systems perform the best relative to the discriminative systems in terms of objective metrics with datasets of 10 h or less. However, their objective metrics performance does not improve when increasing the training dataset size as much as the discriminative systems, and they are outperformed by the discriminative systems with datasets of 100 h or more.