mlsanityData to 14 Oct 2026

Low-Resource Multilingual and Zero-Shot Multispeaker TTS

arXiv 2210.12223 ↗First posted 21 Oct 2022cs.CL cs.SD eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world's over 6,000 spoken languages. In this work, we bring together the tasks of zero-shot voice cloning and multilingual low-resource TTS. Using the language agnostic meta learning (LAML) procedure and modifications to a TTS encoder, we show that it is possible for a system to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language. We show the success of our proposed approach in terms of intelligibility, naturalness and similarity to target speaker using objective metrics as well as human studies and provide our code and trained models open source.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

41 of 56 references.

References · 41

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  8. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  11. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
29 more
  1. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  2. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  3. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  4. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  5. 2018Neural Voice Cloning with a Few Samplescited by 100
  6. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  7. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  8. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  9. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  10. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  11. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  12. 2021One TTS Alignment To Rule Them Allcited by 47
  13. 2020LRSpeech: Extremely Low-Resource Speech Synthesis and Recognitioncited by 43
  14. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  15. 2019End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learningcited by 30
  16. 2020Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embeddingcited by 27
  17. 2021ESPnet2-TTS: Extending the Edge of TTS Researchcited by 22
  18. 2020Phonological Features for 0-shot Multilingual Speech Synthesiscited by 20
  19. 2022Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Featurescited by 20
  20. 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
  21. 2021HUI-Audio-Corpus-German: A high quality TTS datasetcited by 15
  22. 2022Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speechcited by 13
  23. 2021Adapting TTS models For New Speakers using Transfer Learningcited by 12
  24. 2022Speaker Anonymization with Phonetic Intermediate Representationscited by 10
  25. 2020Efficient neural speech synthesis for low-resource languages through multilingual modelingcited by 8
  26. 2021Continual Speaker Adaptation for Text-to-Speech Synthesiscited by 5
  27. 2022PoeticTTS -- Controllable Poetry Reading for Literary Studiescited by 5
  28. 2019IMS-Speech: A Speech to Text Toolcited by 4
  29. 2021VRAIN-UPV MLLP's system for the Blizzard Challenge 2021cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 3; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scalecited by 91
  2. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  3. 2024Meta Learning Text-to-Speech Synthesis in over 7000 Languagescited by 9
  4. 2024Controlling Emotion in Text-to-Speech with Natural Language Promptscited by 9
  5. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  6. 2024Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settingscited by 7
  7. 2023The IMS Toucan System for the Blizzard Challenge 2023cited by 6
  8. 2024Probing the Feasibility of Multilingual Speaker Anonymizationcited by 5
  9. 2024Easy, Interpretable, Effective: openSMILE for voice deepfake detectioncited by 5
  10. 2025Voice Cloning: Comprehensive Surveycited by 4
  11. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  12. 2023Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directionscited by 1
3 more
  1. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimizationcited by 1
  2. 2023An analysis on the effects of speaker embedding choice in non auto-regressive TTScited by 0
  3. 2026What You Train Is What You Get: Gender Bias, Training Composition, and Post-Hoc Mitigation in Audio Deepfake Detectioncited by 0