# Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis

arXiv: https://arxiv.org/abs/2003.06227 · HTML: https://mlsanity.com/paper/2003.06227/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 11 of 11)

- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [Fine-grained style control in Transformer-based Text-to-speech Synthesis](https://mlsanity.com/paper/2110.06306/) (2021)
- [Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models](https://mlsanity.com/paper/2110.02891/) (2021)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [An investigation of phrase break prediction in an End-to-End TTS system](https://mlsanity.com/paper/2304.04157/) (2023)
- [Using multiple reference audios and style embedding constraints for speech synthesis](https://mlsanity.com/paper/2110.04451/) (2021)
- [Learning from Multiple Sources for Data-to-Text and Text-to-Data](https://mlsanity.com/paper/2302.11269/) (2023)
- [DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer](https://mlsanity.com/paper/2305.19567/) (2023)
- [Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement](https://mlsanity.com/paper/2510.01722/) (2025)

## Abstract (arXiv)

We present a method to generate speech from input text and a style vector that is extracted from a reference speech signal in an unsupervised manner, i.e., no style annotation, such as speaker information, is required. Existing unsupervised methods, during training, generate speech by computing style from the corresponding ground truth sample and use a decoder to combine the style vector with the input text. Training the model in such a way leaks content information into the style vector. The decoder can use the leaked content and ignore some of the input text to minimize the reconstruction loss. At inference time, when the reference speech does not match the content input, the output may not contain all of the content of the input text. We refer to this problem as "content leakage", which we address by explicitly estimating and minimizing the mutual information between the style and the content through an adversarial training formulation. We call our method MIST - Mutual Information based Style Content Separation. The main goal of the method is to preserve the input content in the synthesized speech signal, which we measure by the word error rate (WER) and show substantial improvements over state-of-the-art unsupervised speech synthesis methods.