# Using generative modelling to produce varied intonation for speech synthesis

arXiv: https://arxiv.org/abs/1906.04233 · HTML: https://mlsanity.com/paper/1906.04233/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 7 of 7)

- [STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech](https://mlsanity.com/paper/2103.09474/) (2021)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speech](https://mlsanity.com/paper/2110.12539/) (2021)
- [Location, Location: Enhancing the Evaluation of Text-to-Speech Synthesis Using the Rapid Prosody Transcription Paradigm](https://mlsanity.com/paper/2107.02527/) (2021)
- [Comparing normalizing flows and diffusion models for prosody and acoustic modelling in text-to-speech](https://mlsanity.com/paper/2307.16679/) (2023)
- [Controllable Prosody Generation With Partial Inputs](https://mlsanity.com/paper/2303.09446/) (2023)

## Abstract (arXiv)

Unlike human speakers, typical text-to-speech (TTS) systems are unable to produce multiple distinct renditions of a given sentence. This has previously been addressed by adding explicit external control. In contrast, generative models are able to capture a distribution over multiple renditions and thus produce varied renditions using sampling. Typical neural TTS models learn the average of the data because they minimise mean squared error. In the context of prosody, taking the average produces flatter, more boring speech: an "average prosody". A generative model that can synthesise multiple prosodies will, by design, not model average prosody. We use variational autoencoders (VAEs) which explicitly place the most "average" data close to the mean of the Gaussian prior. We propose that by moving towards the tails of the prior distribution, the model will transition towards generating more idiosyncratic, varied renditions. Focusing here on intonation, we investigate the trade-off between naturalness and intonation variation and find that typical acoustic models can either be natural, or varied, but not both. However, sampling from the tails of the VAE prior produces much more varied intonation than the traditional approaches, whilst maintaining the same level of naturalness.