mlsanityData to 7 Oct 2026

Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

arXiv 2104.01409 ↗First posted 3 Apr 2021eess.AS cs.AI cs.SDcited by 156

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Although neural text-to-speech (TTS) models have attracted a lot of attention and succeeded in generating human-like speech, there is still room for improvements to its naturalness and architectural efficiency. In this work, we propose a novel non-autoregressive TTS model, namely Diff-TTS, which achieves highly natural and efficient speech synthesis. Given the text, Diff-TTS exploits a denoising diffusion framework to transform the noise signal into a mel-spectrogram via diffusion time steps. In order to learn the mel-spectrogram distribution conditioned on the text, we present a likelihood-based optimization method for TTS. Furthermore, to boost up the inference speed, we leverage the accelerated sampling method that allows Diff-TTS to generate raw waveforms much faster without significantly degrading perceptual quality. Through experiments, we verified that Diff-TTS generates 28 times faster than the real-time with a single NVIDIA 2080Ti GPU.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 22 of its 31 references that are in the map. First posted on arXiv 3 Apr 2021.

Builds on

22 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2016Layer Normalizationcited by 5,579
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  12. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
9 more
  1. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  2. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  3. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  4. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
  5. 2020End-to-End Adversarial Text-to-Speechcited by 77
  6. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  7. 2020Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesiscited by 54
  8. 2020SpeedySpeech: Efficient Neural Speech Synthesiscited by 15
  9. 2020EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecturecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 29; 2023: 46; 2024: 40; 2025: 21; 2026: 10.

156 papers.

Sort:

By area:

  1. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  2. 2021Score-based Generative Modeling in Latent Spacecited by 634
  3. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  4. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  5. 2022Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equationscited by 273
  6. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  7. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  8. 2022A Survey on Generative Diffusion Modelcited by 238
  9. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  10. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 230
  11. 2021A Survey on Neural Speech Synthesiscited by 182
  12. 2021On Fast Sampling of Diffusion Probabilistic Modelscited by 170
38 more
  1. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  2. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  3. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 147
  4. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  5. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  6. 2022GENIE: Higher-Order Denoising Diffusion Solverscited by 127
  7. 2022Score-Based Generative Models Detect Manifoldscited by 121
  8. 2022Differentially Private Diffusion Modelscited by 118
  9. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillationcited by 115
  10. 2022How to Backdoor Diffusion Models?cited by 102
  11. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 99
  12. 2024Flow map matching with stochastic interpolants: A mathematical framework for consistency modelscited by 99
  13. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  14. 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Priorcited by 78
  15. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  16. 2023Diffusion Models for Black-Box Optimizationcited by 75
  17. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  18. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  19. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  20. 2023StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editingcited by 68
  21. 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Modelscited by 68
  22. 2024Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theorycited by 58
  23. 2021DiffuseMorph: Unsupervised Deformable Image Registration Using Diffusion Modelcited by 57
  24. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Modelscited by 53
  25. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  26. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  27. 2023Class-Balancing Diffusion Modelscited by 47
  28. 2024Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptionscited by 45
  29. 2022Multi-instrument Music Synthesis with Spectrogram Diffusioncited by 42
  30. 2022Categorical SDEs with Simplex Diffusioncited by 42
  31. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
  32. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Datacited by 35
  33. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Modelscited by 34
  34. 2022Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidancecited by 33
  35. 2023Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shiftcited by 31
  36. 2024SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flowcited by 30
  37. 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesiscited by 29
  38. 2021EdiTTS: Score-based Editing for Controllable Text-to-Speechcited by 28

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs
  2. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
  3. 2022ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
  4. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
  5. 2024Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study
  6. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
  7. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
  8. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding