# Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice

arXiv: https://arxiv.org/abs/1812.05253 · HTML: https://mlsanity.com/paper/1812.05253/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2018.

## Built on by (most cited first, 6 of 6)

- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Combining speakers of multiple languages to improve quality of neural voices](https://mlsanity.com/paper/2108.07737/) (2021)
- [Semi-Supervised Learning Based on Reference Model for Low-resource TTS](https://mlsanity.com/paper/2210.14723/) (2022)
- [Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis](https://mlsanity.com/paper/2203.00951/) (2022)
- [AI based Presentation Creator With Customized Audio Content Delivery](https://mlsanity.com/paper/2106.14213/) (2021)

## Abstract (arXiv)

Neural TTS has shown it can generate high quality synthesized speech. In this paper, we investigate the multi-speaker latent space to improve neural TTS for adapting the system to new speakers with only several minutes of speech or enhancing a premium voice by utilizing the data from other speakers for richer contextual coverage and better generalization. A multi-speaker neural TTS model is built with the embedded speaker information in both spectral and speaker latent space. The experimental results show that, with less than 5 minutes of training data from a new speaker, the new model can achieve an MOS score of 4.16 in naturalness and 4.64 in speaker similarity close to human recordings (4.74). For a well-trained premium voice, we can achieve an MOS score of 4.5 for out-of-domain texts, which is comparable to an MOS of 4.58 for professional recordings, and significantly outperforms single speaker result of 4.28.