Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Synthetically generated speech has rapidly approached human levels of naturalness. However, the paradox remains that ASR systems, when trained on TTS output that is judged as natural by humans, continue to perform badly on real speech. In this work, we explore whether this phenomenon is due to the oversmoothing behaviour of models commonly used in TTS, with a particular focus on the behaviour of TTS-for-ASR as the amount of TTS training data is scaled up. We systematically compare Denoising Diffusion Probabilistic Models (DDPM) to Mean Squared Error (MSE) based models for TTS, when used for ASR model training. We test the scalability of the two approaches, varying both the number hours, and the number of different speakers. We find that for a given model size, DDPM can make better use of more data, and a more diverse set of speakers, than MSE models. We achieve the best reported ratio between real and synthetic speech WER to date (1.46), but also find that a large gap remains.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 19 of its 24 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
19 of 24 references.
References · 19
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
- 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
- 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
- 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
- 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
- 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
- 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
- 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
- 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
7 more
- 2019Speech Recognition with Augmented Synthesized Speechcited by 60
- 2019Generating Synthetic Audio Data for Attention-Based Speech Recognition Systemscited by 46
- 2021Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognitioncited by 24
- 2018Training Neural Speech Recognition Systems with Synthetic Speech Augmentationcited by 21
- 2021Less is More: Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identificationcited by 19
- 2022Evaluating and reducing the distance between synthetic and real speech distributionscited by 8
- 2024On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognitioncited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Speaking in Wavelet Domain: A Simple and Efficient Approach to Speed up Speech Diffusion Model
- 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
- 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
- 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
- 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
- 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
- 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
- 2023Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech Synthesis with Found Data