mlsanityData to 7 Oct 2026

An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios

arXiv 2406.08911 ↗First posted 13 Jun 2024cs.CL eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 17 of its 25 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

17 of 25 references.

References · 17

  1. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  3. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  5. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  6. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  7. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  8. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  9. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  10. 2021One TTS Alignment To Rule Them Allcited by 47
  11. 2022WavThruVec: Latent speech representation as intermediate features for neural speech synthesiscited by 17
  12. 2023XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speechcited by 16
5 more
  1. 2023ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representationscited by 15
  2. 2023Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretrainingcited by 10
  3. 2022A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTScited by 9
  4. 2023The VoiceMOS Challenge 2023: Zero-shot Subjective Speech Quality Prediction for Multiple Domainscited by 8
  5. 2023Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Preserving Cultural Identity with Context-Aware Translation Through Multi-Agent AI Systemscited by 2
  2. 2025MultiGen: Child-Friendly Multilingual Speech Generator with LLMscited by 1
  3. 2025BnTTS: Few-Shot Speaker Adaptation in Low-Resource Settingcited by 0
  4. 2026Deterministic Prompting for Speaker-Stable Low-Resource Greek TTScited by 0