# Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder

arXiv: https://arxiv.org/abs/1804.02135 · HTML: https://mlsanity.com/paper/1804.02135/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2018.

## Built on by (most cited first, 22 of 22)

- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit](https://mlsanity.com/paper/2109.06912/) (2021)
- [Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](https://mlsanity.com/paper/2110.02891/) (2021)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [Emphasis control for parallel neural TTS](https://mlsanity.com/paper/2110.03012/) (2021)
- [Unsupervised word-level prosody tagging for controllable speech synthesis](https://mlsanity.com/paper/2202.07200/) (2022)
- [Style Mixture of Experts for Expressive Text-To-Speech Synthesis](https://mlsanity.com/paper/2406.03637/) (2024)
- [Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system](https://mlsanity.com/paper/2103.04097/) (2021)
- [Learning robust speech representation with an articulatory-regularized variational autoencoder](https://mlsanity.com/paper/2104.03204/) (2021)
- [A learned conditional prior for the VAE acoustic space of a TTS system](https://mlsanity.com/paper/2106.10229/) (2021)
- [Do Prosody Transfer Models Transfer Prosody?](https://mlsanity.com/paper/2303.04289/) (2023)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [What Do We See in Them? Identifying Dimensions of Partner Models for Speech Interfaces Using a Psycholexical Approach](https://mlsanity.com/paper/2102.02094/) (2021)
- [Discrete Acoustic Space for an Efficient Sampling in Neural Text-To-Speech](https://mlsanity.com/paper/2110.12539/) (2021)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [Using multiple reference audios and style embedding constraints for speech synthesis](https://mlsanity.com/paper/2110.04451/) (2021)
- [Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech](https://mlsanity.com/paper/2205.04120/) (2022)
- [Affective social anthropomorphic intelligent system](https://mlsanity.com/paper/2304.11046/) (2023)
- [Cross-Utterance Conditioned VAE for Speech Generation](https://mlsanity.com/paper/2309.04156/) (2023)
- [Modeling Sarcastic Speech: Semantic and Prosodic Cues in a Speech Synthesis Framework](https://mlsanity.com/paper/2510.07096/) (2025)
- [Multi-Scale Spectrogram Modelling for Neural Text-to-Speech](https://mlsanity.com/paper/2106.15649/) (2021)

## Abstract (arXiv)

Recent advances in neural autoregressive models have improve the performance of speech synthesis (SS). However, as they lack the ability to model global characteristics of speech (such as speaker individualities or speaking styles), particularly when these characteristics have not been labeled, making neural autoregressive SS systems more expressive is still an open issue. In this paper, we propose to combine VoiceLoop, an autoregressive SS model, with Variational Autoencoder (VAE). This approach, unlike traditional autoregressive SS systems, uses VAE to model the global characteristics explicitly, enabling the expressiveness of the synthesized speech to be controlled in an unsupervised manner. Experiments using the VCTK and Blizzard2012 datasets show the VAE helps VoiceLoop to generate higher quality speech and to control the expressions in its synthesized speech by incorporating global characteristics into the speech generating process.