# Low-Resource Multilingual and Zero-Shot Multispeaker TTS

arXiv: https://arxiv.org/abs/2210.12223 · HTML: https://mlsanity.com/paper/2210.12223/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 56 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 30 of 41)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Hi-Fi Multi-Speaker English TTS Dataset](https://mlsanity.com/paper/2104.01497/) (2021)
- [SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model](https://mlsanity.com/paper/2104.05557/) (2021)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios](https://mlsanity.com/paper/2204.00436/) (2022)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [ESPnet2-TTS: Extending the Edge of TTS Research](https://mlsanity.com/paper/2110.07840/) (2021)
- [Phonological Features for 0-shot Multilingual Speech Synthesis](https://mlsanity.com/paper/2008.04107/) (2020)

## Built on by (most cited first, 15 of 15)

- [ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scale](https://mlsanity.com/paper/2408.08739/) (2024)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [Meta Learning Text-to-Speech Synthesis in over 7000 Languages](https://mlsanity.com/paper/2406.06403/) (2024)
- [Controlling Emotion in Text-to-Speech with Natural Language Prompts](https://mlsanity.com/paper/2406.06406/) (2024)
- [Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation](https://mlsanity.com/paper/2401.09802/) (2024)
- [Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings](https://mlsanity.com/paper/2407.14056/) (2024)
- [The IMS Toucan System for the Blizzard Challenge 2023](https://mlsanity.com/paper/2310.17499/) (2023)
- [Probing the Feasibility of Multilingual Speaker Anonymization](https://mlsanity.com/paper/2407.02937/) (2024)
- [Easy, Interpretable, Effective: openSMILE for voice deepfake detection](https://mlsanity.com/paper/2408.15775/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [Speechless: Speech Instruction Training Without Speech for Low Resource Languages](https://mlsanity.com/paper/2505.17417/) (2025)
- [Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions](https://mlsanity.com/paper/2310.17502/) (2023)
- [Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization](https://mlsanity.com/paper/2509.21718/) (2025)
- [An analysis on the effects of speaker embedding choice in non auto-regressive TTS](https://mlsanity.com/paper/2307.09898/) (2023)
- [What You Train Is What You Get: Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detection](https://mlsanity.com/paper/2607.09891/) (2026)

## Abstract (arXiv)

While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world's over 6,000 spoken languages. In this work, we bring together the tasks of zero-shot voice cloning and multilingual low-resource TTS. Using the language agnostic meta learning (LAML) procedure and modifications to a TTS encoder, we show that it is possible for a system to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language. We show the success of our proposed approach in terms of intelligibility, naturalness and similarity to target speaker using objective metrics as well as human studies and provide our code and trained models open source.