mlsanityData to 14 Oct 2026

XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

arXiv 2406.04904 ↗First posted 7 Jun 2024eess.AS cs.CL cs.SDcited by 179

Not on the map. Why?

Abstract (arXiv)

Most Zero-shot Multi-speaker TTS (ZS-TTS) systems support only a single language. Although models like YourTTS, VALL-E X, Mega-TTS 2, and Voicebox explored Multilingual ZS-TTS they are limited to just a few high/medium resource languages, limiting the applications of these models in most of the low/medium resource languages. In this paper, we aim to alleviate this issue by proposing and making publicly available the XTTS system. Our method builds upon the Tortoise model and adds several novel modifications to enable multilingual training, improve voice cloning, and enable faster training and inference. XTTS was trained in 16 languages and achieved state-of-the-art (SOTA) results in most of them.

Key facts. Cited by 179 papers in the map (data through 14 Oct 2026). Builds on 33 of its 35 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

33 of 35 references.

References · 33

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  9. 2022High Fidelity Neural Audio Compressioncited by 687
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
21 more
  1. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  3. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  4. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  5. 2021A Survey on Neural Speech Synthesiscited by 184
  6. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  7. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
  8. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  9. 2018Neural Voice Cloning with a Few Samplescited by 100
  10. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  11. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  12. 2023Better speech synthesis through scalingcited by 63
  13. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  14. 2023Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesiscited by 41
  15. 2020Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020cited by 36
  16. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  17. 2020Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embeddingcited by 27
  18. 2024ECAPA2: A Hybrid Neural Network Architecture and Training Strategy for Robust Speaker Embeddingscited by 22
  19. 2023CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languagescited by 15
  20. 2022ASR data augmentation in low-resource settings using cross-lingual multi-speaker TTS and cross-lingual voice conversioncited by 10
  21. 2023Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 31; 2025: 72; 2026: 76.

179 papers.

Sort:

By area:

  1. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  2. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  3. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  4. 2024ASVspoof 5: Crowdsourced Speech Data, Deepfakes, and Adversarial Attacks at Scalecited by 91
  5. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  6. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  7. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  8. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 38
  9. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 36
  10. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32
  11. 2025IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech Systemcited by 28
  12. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
38 more
  1. 2026OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Modelscited by 25
  2. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  3. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  4. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  5. 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 15
  6. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  7. 2024I Can Hear You: Selective Robust Training for Deepfake Audio Detectioncited by 13
  8. 2025CS-FLEURS: A Massively Multilingual and Code-Switched Speech Datasetcited by 12
  9. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
  10. 2024Where are we in audio deepfake detection? A systematic analysis over generative and detection modelscited by 11
  11. 2024Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognitioncited by 9
  12. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  13. 2025DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Datasetcited by 8
  14. 2025TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systemscited by 8
  15. 2025AV-Deepfake1M++: A Large-Scale Audio-Visual Deepfake Benchmark with Real-World Perturbationscited by 8
  16. 2025LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLMcited by 7
  17. 2025UITron-Speech: Towards Automated GUI Agents Based on Speech Instructionscited by 7
  18. 2025Asynchronous Reasoning: Training-Free Interactive Thinking LLMscited by 7
  19. 2025SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesiscited by 6
  20. 2025Tell me Habibi, is it Real or Fake?cited by 6
  21. 2025SCDF: A Speaker Characteristics DeepFake Speech Dataset for Bias Analysiscited by 6
  22. 2026IndexTTS 2.5 Technical Reportcited by 6
  23. 2024Voice Disorder Analysis: a Transformer-based Approachcited by 5
  24. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  25. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusioncited by 5
  26. 2025VoXtream: Full-Stream Text-to-Speech with Extremely Low Latencycited by 5
  27. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  28. 2026TRACE: Training-Free Partial Audio Deepfake Detection via Embedding Trajectory Analysis of Speech Foundation Modelscited by 5
  29. 2024SF-Speech: Straightened Flow for Zero-Shot Voice Clonecited by 4
  30. 2025Voice Cloning: Comprehensive Surveycited by 4
  31. 2025Pairwise Evaluation of Accent Similarity in Speech Synthesiscited by 4
  32. 2025Synthetic Speech Source Tracing using Metric Learningcited by 4
  33. 2025SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agentscited by 4
  34. 2025The State Of TTS: A Case Study with Human Fooling Ratescited by 4
  35. 2026Towards Robust Speech Deepfake Detection via Human-Inspired Reasoningcited by 4
  36. 2024kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speechcited by 3
  37. 2024A Framework for Synthetic Audio Conversations Generation using Large Language Modelscited by 3
  38. 2025Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancementcited by 3

50 most cited of 179 All 179 (JSON)