# Mix and Match: An Empirical Study on Training Corpus Composition for Polyglot Text-To-Speech (TTS)

arXiv: https://arxiv.org/abs/2207.01507 · HTML: https://mlsanity.com/paper/2207.01507/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Improving Variational Auto-Encoders using Householder Flow](https://mlsanity.com/paper/1611.09630/) (2016)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018)
- [Accent and Speaker Disentanglement in Many-to-many Voice Conversion](https://mlsanity.com/paper/2011.08609/) (2020)
- [Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversion](https://mlsanity.com/paper/2010.08136/) (2020)
- [Unsupervised Polyglot Text To Speech](https://mlsanity.com/paper/1902.02263/) (2019)
- [Cross-lingual Low Resource Speaker Adaptation Using Phonological Features](https://mlsanity.com/paper/2111.09075/) (2021)
- [Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training](https://mlsanity.com/paper/2201.08124/) (2022)
- [Exploring Disentanglement with Multilingual and Monolingual VQ-VAE](https://mlsanity.com/paper/2105.01573/) (2021)
- [Comprehensive evaluation of statistical speech waveform synthesis](https://mlsanity.com/paper/1811.06296/) (2018)

## Built on by (most cited first, 4 of 4)

- [VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing](https://mlsanity.com/paper/2404.06674/) (2024)
- [Modelling low-resource accents without accent-specific TTS frontend](https://mlsanity.com/paper/2301.04606/) (2023)
- [Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech](https://mlsanity.com/paper/2309.08255/) (2023)
- [Unseen Speaker and Language Adaptation for Lightweight Text-To-Speech with Adapters](https://mlsanity.com/paper/2508.18006/) (2025)

## Abstract (arXiv)

Training multilingual Neural Text-To-Speech (NTTS) models using only monolingual corpora has emerged as a popular way for building voice cloning based Polyglot NTTS systems. In order to train these models, it is essential to understand how the composition of the training corpora affects the quality of multilingual speech synthesis. In this context, it is common to hear questions such as "Would including more Spanish data help my Italian synthesis, given the closeness of both languages?". Unfortunately, we found existing literature on the topic lacking in completeness in this regard. In the present work, we conduct an extensive ablation study aimed at understanding how various factors of the training corpora, such as language family affiliation, gender composition, and the number of speakers, contribute to the quality of Polyglot synthesis. Our findings include the observation that female speaker data are preferred in most scenarios, and that it is not always beneficial to have more speakers from the target language variant in the training corpus. The findings herein are informative for the process of data procurement and corpora building.