# Deep Voice 2: Multi-Speaker Neural Text-to-Speech

arXiv: https://arxiv.org/abs/1705.08947 · HTML: https://mlsanity.com/paper/1705.08947/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2017.

## Built on by (most cited first, 30 of 75)

- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation](https://mlsanity.com/paper/2106.03153/) (2021)
- [Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forward](https://mlsanity.com/paper/2103.00484/) (2021)
- [WaveFake: A Data Set to Facilitate Audio Deepfake Detection](https://mlsanity.com/paper/2111.02813/) (2021)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs](https://mlsanity.com/paper/2201.11972/) (2022)
- [AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios](https://mlsanity.com/paper/2204.00436/) (2022)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Towards Building Text-To-Speech Systems for the Next Billion Users](https://mlsanity.com/paper/2211.09536/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit](https://mlsanity.com/paper/2109.06912/) (2021)
- [V2C: Visual Voice Cloning](https://mlsanity.com/paper/2111.12890/) (2021)
- [Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech](https://mlsanity.com/paper/2111.04040/) (2021)
- [The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning](https://mlsanity.com/paper/2406.04328/) (2024)
- [GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis](https://mlsanity.com/paper/2106.15153/) (2021)
- [Detecting Individual Decision-Making Style: Exploring Behavioral Stylometry in Chess](https://mlsanity.com/paper/2208.01366/) (2022)
- [SpeechNet: A Universal Modularized Model for Speech Processing Tasks](https://mlsanity.com/paper/2105.03070/) (2021)
- [Low-Resource Multilingual and Zero-Shot Multispeaker TTS](https://mlsanity.com/paper/2210.12223/) (2022)
- [Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech](https://mlsanity.com/paper/2106.12896/) (2021)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention](https://mlsanity.com/paper/2102.06431/) (2021)
- [Adapting TTS models For New Speakers using Transfer Learning](https://mlsanity.com/paper/2110.05798/) (2021)

## Abstract (arXiv)

We introduce a technique for augmenting neural text-to-speech (TTS) with lowdimensional trainable speaker embeddings to generate different voices from a single model. As a starting point, we show improvements over the two state-ofthe-art approaches for single-speaker neural TTS: Deep Voice 1 and Tacotron. We introduce Deep Voice 2, which is based on a similar pipeline with Deep Voice 1, but constructed with higher performance building blocks and demonstrates a significant audio quality improvement over Deep Voice 1. We improve Tacotron by introducing a post-processing neural vocoder, and demonstrate a significant audio quality improvement. We then demonstrate our technique for multi-speaker speech synthesis for both Deep Voice 2 and Tacotron on two multi-speaker TTS datasets. We show that a single neural TTS system can learn hundreds of unique voices from less than half an hour of data per speaker, while achieving high audio quality synthesis and preserving the speaker identities almost perfectly.