# A multilingual training strategy for low resource Text to Speech

arXiv: https://arxiv.org/abs/2409.01217 · HTML: https://mlsanity.com/paper/2409.01217/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [Listening while Speaking: Speech Chain by Deep Learning](https://mlsanity.com/paper/1707.04879/) (2017)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [Almost Unsupervised Text to Speech and Automatic Speech Recognition](https://mlsanity.com/paper/1905.06791/) (2019)
- [The Zero Resource Speech Challenge 2019: TTS without T](https://mlsanity.com/paper/1904.11469/) (2019)
- [BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpus](https://mlsanity.com/paper/2207.03546/) (2022)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [Exploring Transfer Learning for Low Resource Emotional TTS](https://mlsanity.com/paper/1901.04276/) (2019)
- [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018)
- [Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning](https://mlsanity.com/paper/1910.12729/) (2019)
- [Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages](https://mlsanity.com/paper/2008.04549/) (2020)
- [CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages](https://mlsanity.com/paper/2306.10097/) (2023)
- [Neural Speech Synthesis with Transformer Network](https://mlsanity.com/paper/1809.08895/) (2018)
- [Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition](https://mlsanity.com/paper/2203.15796/) (2022)

## Built on by (most cited first, 3 of 3)

- [Large Multimodal Models for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2502.05568/) (2025)
- [ELAICHI: Enhancing Low-resource TTS by Addressing Infrequent and Low-frequency Character Bigrams](https://mlsanity.com/paper/2410.17901/) (2024)
- [Deterministic Prompting for Speaker-Stable Low-Resource Greek TTS](https://mlsanity.com/paper/2609.10022/) (2026)

## Abstract (arXiv)

Recent speech technologies have led to produce high quality synthesised speech due to recent advances in neural Text to Speech (TTS). However, such TTS models depend on extensive amounts of data that can be costly to produce and is hardly scalable to all existing languages, especially that seldom attention is given to low resource languages. With techniques such as knowledge transfer, the burden of creating datasets can be alleviated. In this paper, we therefore investigate two aspects; firstly, whether data from social media can be used for a small TTS dataset construction, and secondly whether cross lingual transfer learning (TL) for a low resource language can work with this type of data. In this aspect, we specifically assess to what extent multilingual modeling can be leveraged as an alternative to training on monolingual corporas. To do so, we explore how data from foreign languages may be selected and pooled to train a TTS model for a target low resource language. Our findings show that multilingual pre-training is better than monolingual pre-training at increasing the intelligibility and naturalness of the generated speech.