# Semi-Supervised Generative Modeling for Controllable Speech Synthesis

arXiv: https://arxiv.org/abs/1910.01709 · HTML: https://mlsanity.com/paper/1910.01709/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2019.

## Built on by (most cited first, 21 of 21)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [Controllable Data Generation by Deep Learning: A Review](https://mlsanity.com/paper/2207.09542/) (2022)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [Emotional Prosody Control for Speech Generation](https://mlsanity.com/paper/2111.04730/) (2021)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Unsupervised word-level prosody tagging for controllable speech synthesis](https://mlsanity.com/paper/2202.07200/) (2022)
- [Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesis](https://mlsanity.com/paper/2212.06397/) (2022)
- [Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis](https://mlsanity.com/paper/2303.07711/) (2023)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [Sound Check: Auditing Audio Datasets](https://mlsanity.com/paper/2410.13114/) (2024)
- [Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder](https://mlsanity.com/paper/2212.08329/) (2022)
- [UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2505.10599/) (2025)
- [ReCAB-VAE: Gumbel-Softmax Variational Inference Based on Analytic Divergence](https://mlsanity.com/paper/2205.04104/) (2022)
- [Challenging the Semi-Supervised VAE Framework for Text Classification](https://mlsanity.com/paper/2109.12969/) (2021)
- [Semi-Supervised Junction Tree Variational Autoencoder for Molecular Property Prediction](https://mlsanity.com/paper/2208.05119/) (2022)
- [Iterative Self-Learning for Expressive Text-to-Speech Synthesis](https://mlsanity.com/paper/2608.15910/) (2026)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)

## Abstract (arXiv)

We present a novel generative model that combines state-of-the-art neural text-to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. Audio samples are available on the web.