# Combining speakers of multiple languages to improve quality of neural voices

arXiv: https://arxiv.org/abs/2108.07737 · HTML: https://mlsanity.com/paper/2108.07737/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS](https://mlsanity.com/paper/1906.00672/) (2019)
- [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018)
- [Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions](https://mlsanity.com/paper/2008.05289/) (2020)
- [Building a mixed-lingual neural TTS system with only monolingual data](https://mlsanity.com/paper/1904.06063/) (2019)
- [Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS?](https://mlsanity.com/paper/2005.01245/) (2020)
- [Training Multi-Speaker Neural Text-to-Speech Systems using Speaker-Imbalanced Speech Corpora](https://mlsanity.com/paper/1904.00771/) (2019)
- [Generating Multilingual Voices Using Speaker Space Translation Based on Bilingual Speaker Data](https://mlsanity.com/paper/2004.04972/) (2020)
- [Modeling Multi-speaker Latent Space to Improve Neural TTS: Quick Enrolling New Speaker and Enhancing Premium Voice](https://mlsanity.com/paper/1812.05253/) (2018)
- [Multi-Speaker End-to-End Speech Synthesis](https://mlsanity.com/paper/1907.04462/) (2019)

## Built on by (most cited first, 3 of 3)

- [Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01547/) (2022)
- [Data-augmented cross-lingual synthesis in a teacher-student framework](https://mlsanity.com/paper/2204.00061/) (2022)
- [Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech](https://mlsanity.com/paper/2309.08255/) (2023)

## Abstract (arXiv)

In this work, we explore multiple architectures and training procedures for developing a multi-speaker and multi-lingual neural TTS system with the goals of a) improving the quality when the available data in the target language is limited and b) enabling cross-lingual synthesis. We report results from a large experiment using 30 speakers in 8 different languages across 15 different locales. The system is trained on the same amount of data per speaker. Compared to a single-speaker model, when the suggested system is fine tuned to a speaker, it produces significantly better quality in most of the cases while it only uses less than 40% of the speaker's data used to build the single-speaker model. In cross-lingual synthesis, on average, the generated quality is within 80% of native single-speaker models, in terms of Mean Opinion Score.