# One Model, Many Languages: Meta-learning for Multilingual Text-to-Speech

arXiv: https://arxiv.org/abs/2008.00768 · HTML: https://mlsanity.com/paper/2008.00768/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2020.

## Built on by (most cited first, 28 of 28)

- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis](https://mlsanity.com/paper/2103.03541/) (2021)
- [Heterogeneous Target Speech Separation](https://mlsanity.com/paper/2204.03594/) (2022)
- [Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2308.01831/) (2023)
- [Cross-lingual Low Resource Speaker Adaptation Using Phonological Features](https://mlsanity.com/paper/2111.09075/) (2021)
- [Towards Quantifying and Reducing Language Mismatch Effects in Cross-Lingual Speech Anti-Spoofing](https://mlsanity.com/paper/2409.08346/) (2024)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [Continual Speaker Adaptation for Text-to-Speech Synthesis](https://mlsanity.com/paper/2103.14512/) (2021)
- [Multilingual Speech-to-Speech Translation into Multiple Target Languages](https://mlsanity.com/paper/2307.08655/) (2023)
- [Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction](https://mlsanity.com/paper/2312.10305/) (2023)
- [Revisiting IPA-based Cross-lingual Text-to-speech](https://mlsanity.com/paper/2110.07187/) (2021)
- [ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations](https://mlsanity.com/paper/2303.01261/) (2023)
- [X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning](https://mlsanity.com/paper/2605.05611/) (2026)
- [Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation](https://mlsanity.com/paper/2210.17264/) (2022)
- [Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis](https://mlsanity.com/paper/2110.04482/) (2021)
- [DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarin](https://mlsanity.com/paper/2309.00883/) (2023)
- [BiSinger: Bilingual Singing Voice Synthesis](https://mlsanity.com/paper/2309.14089/) (2023)
- [Improve Cross-lingual Voice Cloning Using Low-quality Code-switched Data](https://mlsanity.com/paper/2110.07210/) (2021)
- [Data-augmented cross-lingual synthesis in a teacher-student framework](https://mlsanity.com/paper/2204.00061/) (2022)
- [SANE-TTS: Stable And Natural End-to-End Multilingual Text-to-Speech](https://mlsanity.com/paper/2206.12132/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)
- [MParrotTTS: Multilingual Multi-speaker Text to Speech Synthesis in Low Resource Setting](https://mlsanity.com/paper/2305.11926/) (2023)
- [PRESENT: Zero-Shot Text-to-Prosody Control](https://mlsanity.com/paper/2408.06827/) (2024)

## Abstract (arXiv)

We introduce an approach to multilingual speech synthesis which uses the meta-learning concept of contextual parameter generation and produces natural-sounding multilingual speech using more languages and less training data than previous approaches. Our model is based on Tacotron 2 with a fully convolutional input text encoder whose weights are predicted by a separate parameter generator network. To boost voice cloning, the model uses an adversarial speaker classifier with a gradient reversal layer that removes speaker-specific information from the encoder. We arranged two experiments to compare our model with baselines using various levels of cross-lingual parameter sharing, in order to evaluate: (1) stability and performance when training on low amounts of data, (2) pronunciation accuracy and voice quality of code-switching synthesis. For training, we used the CSS10 dataset and our new small dataset based on Common Voice recordings in five languages. Our model is shown to effectively share information across languages and according to a subjective evaluation test, it produces more natural and accurate code-switching speech than the baselines.