mlsanityData to 14 Oct 2026

Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech

arXiv 2210.15447 ↗First posted 27 Oct 2022cs.SD cs.CL eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

This paper proposes Virtuoso, a massively multilingual speech-text joint semi-supervised learning framework for text-to-speech synthesis (TTS) models. Existing multilingual TTS typically supports tens of languages, which are a small fraction of the thousands of languages in the world. One difficulty to scale multilingual TTS to hundreds of languages is collecting high-quality speech-text paired data in low-resource languages. This study extends Maestro, a speech-text joint pretraining framework for automatic speech recognition (ASR), to speech generation tasks. To train a TTS model from various types of speech and text data, different training schemes are designed to handle supervised (paired TTS and ASR data) and unsupervised (untranscribed speech and unspoken text) datasets. Experimental evaluation shows that 1) multilingual TTS models trained on Virtuoso can achieve significantly better naturalness and intelligibility than baseline ones in seen languages, and 2) they can synthesize reasonably intelligible and naturally sounding speech for unseen languages where no high-quality paired TTS data is available.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

31 of 41 references.

References · 31

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  7. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
19 more
  1. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  2. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  3. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  4. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  5. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  6. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  7. 2018Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytescited by 54
  8. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  9. 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
  10. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  11. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  12. 2022Building African Voicescited by 14
  13. 2020JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignmentcited by 13
  14. 2021Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesiscited by 13
  15. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  16. 2020Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversioncited by 12
  17. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10
  18. 2022Speaker consistency loss and step-wise optimization for semi-supervised joint training of TTS and ASR using unpaired text datacited by 3
  19. 2021Towards Lifelong Learning of Multilingual Text-To-Speech Synthesiscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 1; 2026: 1.

15 papers.

Sort:

  1. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  2. 2023Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretrainingcited by 10
  3. 2024Meta Learning Text-to-Speech Synthesis in over 7000 Languagescited by 9
  4. 2024Utilizing TTS Synthesized Data for Efficient Development of Keyword Spotting Modelcited by 4
  5. 2024Multilingual Prosody Transfer: Comparing Supervised & Transfer Learningcited by 2
  6. 2024ASTRA: Aligning Speech and Text Representations for Asr without Samplingcited by 2
  7. 2024Synth4Kws: Synthesized Speech for User Defined Keyword Spotting in Low Resource Environmentscited by 2
  8. 2025RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyan Code-Switched Datasetcited by 2
  9. 2023Few-Shot Spoken Language Understanding via Joint Speech-Text Modelscited by 1
  10. 2024wav2graph: A Framework for Supervised Learning Knowledge Graph from Speechcited by 1
  11. 2024Adversarial training of Keyword Spotting to Minimize TTS Data Overfittingcited by 1
  12. 2024Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitationcited by 1
3 more
  1. 2024Can DeepFake Speech be Reliably Detected?cited by 1
  2. 2026ChemPro: A Progressive Chemistry Benchmark for Large Language Modelscited by 1
  3. 2024Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initializationcited by 0