# Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis

arXiv: https://arxiv.org/abs/1906.03402 · HTML: https://mlsanity.com/paper/1906.03402/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2019.

## Built on by (most cited first, 15 of 15)

- [GAN Vocoder: Multi-Resolution Discriminator Is All You Need](https://mlsanity.com/paper/2103.05236/) (2021)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [Emotional Prosody Control for Speech Generation](https://mlsanity.com/paper/2111.04730/) (2021)
- [Learning De-identified Representations of Prosody from Raw Audio](https://mlsanity.com/paper/2107.08248/) (2021)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Fast DCTTS: Efficient Deep Convolutional Text-to-Speech](https://mlsanity.com/paper/2104.00624/) (2021)
- [Do Prosody Transfer Models Transfer Prosody?](https://mlsanity.com/paper/2303.04289/) (2023)
- [On Prosody Modeling for ASR+TTS based Voice Conversion](https://mlsanity.com/paper/2107.09477/) (2021)
- [Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.10168/) (2021)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [GANtron: Emotional Speech Synthesis with Generative Adversarial Networks](https://mlsanity.com/paper/2110.03390/) (2021)
- [Zero-Shot Long-Form Voice Cloning with Dynamic Convolution Attention](https://mlsanity.com/paper/2201.10375/) (2022)
- [Into-TTS : Intonation Template Based Prosody Control System](https://mlsanity.com/paper/2204.01271/) (2022)
- [An investigation of phrase break prediction in an End-to-End TTS system](https://mlsanity.com/paper/2304.04157/) (2023)
- [Controllable Speaking Styles Using a Large Language Model](https://mlsanity.com/paper/2305.10321/) (2023)

## Abstract (arXiv)

Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis (supporting control and transfer of prosody and style), but has not presented a coherent framework for understanding the trade-offs between the competing methods. In this paper, we propose embedding capacity (the amount of information the embedding contains about the data) as a unified method of analyzing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information. In our proposed model (Capacitron), we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior, the same model can be used for high-precision prosody transfer, text-agnostic style transfer, and generation of natural-sounding prior samples. For multi-speaker models, Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior. Lastly, we introduce a method for decomposing embedding capacity hierarchically across two sets of latents, allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior. Audio examples are available on the web.