# Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis

arXiv: https://arxiv.org/abs/1807.11470 · HTML: https://mlsanity.com/paper/1807.11470/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2018.

## Built on by (most cited first, 13 of 13)

- [Vector Quantized Wasserstein Auto-Encoder](https://mlsanity.com/paper/2302.05917/) (2023)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis](https://mlsanity.com/paper/2306.09417/) (2023)
- [Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](https://mlsanity.com/paper/2110.02891/) (2021)
- [Multi-objective Deep Data Generation with Correlated Property Control](https://mlsanity.com/paper/2210.01796/) (2022)
- [Audio representations for deep learning in sound synthesis: A review](https://mlsanity.com/paper/2201.02490/) (2022)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Autoregressive Co-Training for Learning Discrete Speech Representations](https://mlsanity.com/paper/2203.15840/) (2022)
- [Analysis and Assessment of Controllability of an Expressive Deep Learning-based TTS system](https://mlsanity.com/paper/2103.04097/) (2021)
- [Reproducible, incremental representation learning with Rosetta VAE](https://mlsanity.com/paper/2201.05206/) (2022)
- [Particle physics DL-simulation with control over generated data properties](https://mlsanity.com/paper/2405.14049/) (2024)
- [Controllable Data Generation Via Iterative Data-Property Mutual Mappings](https://mlsanity.com/paper/2310.07683/) (2023)
- [A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer](https://mlsanity.com/paper/2406.06601/) (2024)

## Abstract (arXiv)

Generating versatile and appropriate synthetic speech requires control over the output expression separate from the spoken text. Important non-textual speech variation is seldom annotated, in which case output control must be learned in an unsupervised fashion. In this paper, we perform an in-depth study of methods for unsupervised learning of control in statistical speech synthesis. For example, we show that popular unsupervised training heuristics can be interpreted as variational inference in certain autoencoder models. We additionally connect these models to VQ-VAEs, another, recently-proposed class of deep variational autoencoders, which we show can be derived from a very similar mathematical argument. The implications of these new probabilistic interpretations are discussed. We illustrate the utility of the various approaches with an application to acoustic modelling for emotional speech synthesis, where the unsupervised methods for learning expression control (without access to emotional labels) are found to give results that in many aspects match or surpass the previous best supervised approach.