mlsanityData to 7 Oct 2026

Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining

arXiv 2301.12596 ↗First posted 30 Jan 2023eess.AS cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 37 of its 68 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

37 of 68 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Layer Normalizationcited by 5,579
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
25 more
  1. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  3. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  5. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  6. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  7. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  8. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  9. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  10. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  11. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  12. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  13. 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
  14. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  15. 2022ASR2K: Speech Recognition for Around 2000 Languages without Audiocited by 26
  16. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  17. 2020Phonological Features for 0-shot Multilingual Speech Synthesiscited by 20
  18. 2022Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Featurescited by 20
  19. 2014BilBOWA: Fast Bilingual Distributed Representations without Word Alignmentscited by 16
  20. 2020Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languagescited by 16
  21. 2022Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speechcited by 15
  22. 2022Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognitioncited by 14
  23. 2021Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesiscited by 13
  24. 2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13
  25. 2022Text-to-speech synthesis from dark data with evaluation-in-the-loop data selectioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 2; 2026: 1.

10 papers.

  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  2. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  3. 2026Towards Explicit Acoustic Evidence Perception in Audio LLMs for Speech Deepfake Detectioncited by 5
  4. 2024An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarioscited by 4
  5. 2024Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptationcited by 1
  6. 2024Zero-Shot Text-to-Speech as Golden Speech Generator: A Systematic Framework and its Applicability in Automatic Pronunciation Assessmentcited by 1
  7. 2025SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech Systemcited by 1
  8. 2025BanglaFake: Constructing and Evaluating a Specialized Bengali Deepfake Audio Datasetcited by 1
  9. 2024Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initializationcited by 0
  10. 2024PRESENT: Zero-Shot Text-to-Prosody Controlcited by 0