# Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling

arXiv: https://arxiv.org/abs/2211.10568 · HTML: https://mlsanity.com/paper/2211.10568/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2022.

## Built on by (most cited first, 3 of 3)

- [EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion](https://mlsanity.com/paper/2411.16726/) (2024)
- [Marco-Voice Technical Report](https://mlsanity.com/paper/2508.02038/) (2025)
- [Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness](https://mlsanity.com/paper/2404.06714/) (2024)

## Abstract (arXiv)

This paper aims to synthesize the target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers. We address this challenging problem with a two-stage framework composed of a text-to-style-and-emotion (Text2SE) module and a style-and-emotion-to-wave (SE2Wave) module, bridging by neural bottleneck (BN) features. To further solve the multi-factor (speaker timbre, speaking style and emotion) decoupling problem, we adopt the multi-label binary vector (MBV) and mutual information (MI) minimization to respectively discretize the extracted embeddings and disentangle these highly entangled factors in both Text2SE and SE2Wave modules. Moreover, we introduce a semi-supervised training strategy to leverage data from multiple speakers, including emotion-labeled data, style-labeled data, and unlabeled data. To better transfer the fine-grained expression from references to the target speaker in non-parallel transfer, we introduce a reference-candidate pool and propose an attention-based reference selection approach. Extensive experiments demonstrate the good design of our model.