# Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

arXiv: https://arxiv.org/abs/2105.06337 · HTML: https://mlsanity.com/paper/2105.06337/

## Key facts

- Cited by 447 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 13 May 2021.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Permutation Invariant Graph Generation via Score-Based Generative Modeling](https://mlsanity.com/paper/2003.00638/) (2020)
- [Learning Gradient Fields for Shape Generation](https://mlsanity.com/paper/2008.06520/) (2020)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [Diffusion models for Handwriting Generation](https://mlsanity.com/paper/2011.06704/) (2020)

## Built on by (most cited first, 30 of 447)

- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling](https://mlsanity.com/paper/2106.01357/) (2021)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models](https://mlsanity.com/paper/2404.07724/) (2024)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization](https://mlsanity.com/paper/2308.03686/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale](https://mlsanity.com/paper/2303.06555/) (2023)
- [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion](https://mlsanity.com/paper/2203.13777/) (2022)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)

## Abstract (arXiv)

Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.