# Using VAEs and Normalizing Flows for One-shot Text-To-Speech Synthesis of Expressive Speech

arXiv: https://arxiv.org/abs/1911.12760 · HTML: https://mlsanity.com/paper/1911.12760/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 8 of 8)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model](https://mlsanity.com/paper/2106.09317/) (2021)
- [PSRFlow: Probabilistic Super Resolution with Flow-Based Models for Scientific Data](https://mlsanity.com/paper/2308.04605/) (2023)
- [Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation](https://mlsanity.com/paper/2411.12719/) (2024)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [Distribution augmentation for low-resource expressive text-to-speech](https://mlsanity.com/paper/2202.06409/) (2022)
- [An investigation of phrase break prediction in an End-to-End TTS system](https://mlsanity.com/paper/2304.04157/) (2023)

## Abstract (arXiv)

We propose a Text-to-Speech method to create an unseen expressive style using one utterance of expressive speech of around one second. Specifically, we enhance the disentanglement capabilities of a state-of-the-art sequence-to-sequence based system with a Variational AutoEncoder (VAE) and a Householder Flow. The proposed system provides a 22% KL-divergence reduction while jointly improving perceptual metrics over state-of-the-art. At synthesis time we use one example of expressive style as a reference input to the encoder for generating any text in the desired style. Perceptual MUSHRA evaluations show that we can create a voice with a 9% relative naturalness improvement over standard Neural Text-to-Speech, while also improving the perceived emotional intensity (59 compared to the 55 of neutral speech).