# An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios

arXiv: https://arxiv.org/abs/2406.08911 · HTML: https://mlsanity.com/paper/2406.08911/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [WavThruVec: Latent speech representation as intermediate features for neural speech synthesis](https://mlsanity.com/paper/2203.16930/) (2022)
- [XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech](https://mlsanity.com/paper/2305.19709/) (2023)
- [ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representations](https://mlsanity.com/paper/2312.14398/) (2023)
- [Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining](https://mlsanity.com/paper/2301.12596/) (2023)
- [A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS](https://mlsanity.com/paper/2209.10887/) (2022)
- [The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality Prediction for Multiple Domains](https://mlsanity.com/paper/2310.02640/) (2023)
- [Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selection](https://mlsanity.com/paper/2306.12040/) (2023)

## Built on by (most cited first, 4 of 4)

- [Preserving Cultural Identity with Context-Aware Translation Through Multi-Agent AI Systems](https://mlsanity.com/paper/2503.04827/) (2025)
- [MultiGen: Child-Friendly Multilingual Speech Generator with LLMs](https://mlsanity.com/paper/2508.08715/) (2025)
- [BnTTS: Few-Shot Speaker Adaptation in Low-Resource Setting](https://mlsanity.com/paper/2502.05729/) (2025)
- [Deterministic Prompting for Speaker-Stable Low-Resource Greek TTS](https://mlsanity.com/paper/2609.10022/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS.