# Comparing normalizing flows and diffusion models for prosody and acoustic modelling in text-to-speech

arXiv: https://arxiv.org/abs/2307.16679 · HTML: https://mlsanity.com/paper/2307.16679/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 20 of 20)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Diff-TTS: A Denoising Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2104.01409/) (2021)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network](https://mlsanity.com/paper/1905.07195/) (2019)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [iEmoTTS: Toward Robust Cross-Speaker Emotion Transfer and Control for Speech Synthesis based on Disentanglement between Prosody and Timbre](https://mlsanity.com/paper/2206.14866/) (2022)
- [Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech](https://mlsanity.com/paper/2106.12896/) (2021)
- [Text-free non-parallel many-to-many voice conversion using normalising flows](https://mlsanity.com/paper/2203.08009/) (2022)
- [Using generative modelling to produce varied intonation for speech synthesis](https://mlsanity.com/paper/1906.04233/) (2019)
- [Expressive, Variable, and Controllable Duration Modelling in TTS](https://mlsanity.com/paper/2206.14165/) (2022)
- [Reducing over-smoothness in speech synthesis using Generative Adversarial Networks](https://mlsanity.com/paper/1810.10989/) (2018)
- [Remap, warp and attend: Non-parallel many-to-many accent conversion with Normalizing Flows](https://mlsanity.com/paper/2211.05850/) (2022)

## Built on by (most cited first, 2 of 2)

- [An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR](https://mlsanity.com/paper/2503.08954/) (2025)
- [On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition](https://mlsanity.com/paper/2407.21476/) (2024)

## Abstract (arXiv)

Neural text-to-speech systems are often optimized on L1/L2 losses, which make strong assumptions about the distributions of the target data space. Aiming to improve those assumptions, Normalizing Flows and Diffusion Probabilistic Models were recently proposed as alternatives. In this paper, we compare traditional L1/L2-based approaches to diffusion and flow-based approaches for the tasks of prosody and mel-spectrogram prediction for text-to-speech synthesis. We use a prosody model to generate log-f0 and duration features, which are used to condition an acoustic model that generates mel-spectrograms. Experimental results demonstrate that the flow-based model achieves the best performance for spectrogram prediction, improving over equivalent diffusion and L1 models. Meanwhile, both diffusion and flow-based prosody predictors result in significant improvements over a typical L2-trained prosody models.