mlsanityData to 7 Oct 2026

Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation

arXiv 2409.09311 ↗First posted 14 Sep 2024eess.AS cs.SDcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Diffusion models have achieved remarkable success in text-to-speech (TTS), even in zero-shot scenarios. Recent efforts aim to address the trade-off between inference speed and sound quality, often considered the primary drawback of diffusion models. However, we find a critical mispronunciation issue is being overlooked. Our preliminary study reveals the unstable pronunciation resulting from the diffusion process. Based on this observation, we introduce StableForm-TTS, a novel zero-shot speech synthesis framework designed to produce robust pronunciation while maintaining the advantages of diffusion modeling. By pioneering the adoption of source-filter theory in diffusion TTS, we propose an elaborate architecture for stable formant generation. Experimental results on unseen speakers show that our model outperforms the state-of-the-art method in terms of pronunciation accuracy and naturalness, with comparable speaker similarity. Moreover, our model demonstrates effective scalability as both data and model sizes increase. Audio samples are available online: https://deepbrainai-research.github.io/stableformtts/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 38 of its 49 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

38 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech31 Aug
  2. 2023Better speech synthesis through scaling12 May
  3. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  4. 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech30 Dec
  5. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Models17 Nov
  6. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
  7. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep
  8. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  9. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  9. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  10. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
16 more
  1. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  2. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  3. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  4. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  5. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
  6. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  7. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
  8. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  9. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  10. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  11. 2022HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancementcited by 49
  12. 2021One TTS Alignment To Rule Them Allcited by 47
  13. 2023Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesiscited by 41
  14. 2022Revisiting Over-Smoothness in Text to Speechcited by 31
  15. 2021FastPitchFormant: Source-filter based Decomposed Modeling for Speech Synthesiscited by 12
  16. 2023A Study on Incorporating Whisper for Robust Speech Assessmentcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
  2. 2024Zero-shot Voice Conversion with Diffusion Transformers
  3. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesis
  4. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis
  5. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  6. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluator
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  8. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis