mlsanityData to 14 Oct 2026

BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data

arXiv 2402.08093 ↗First posted 12 Feb 2024cs.LG cs.CL eess.AScited by 75

Not on the map. Why?

Abstract (arXiv)

We introduce a text-to-speech (TTS) model called BASE TTS, which stands for Big Adaptive Streamable TTS with Emergent abilities. BASE TTS is the largest TTS model to-date, trained on 100K hours of public domain speech data, achieving a new state-of-the-art in speech naturalness. It deploys a 1-billion-parameter autoregressive Transformer that converts raw texts into discrete codes ("speechcodes") followed by a convolution-based decoder which converts these speechcodes into waveforms in an incremental, streamable manner. Further, our speechcodes are built using a novel speech tokenization technique that features speaker ID disentanglement and compression with byte-pair encoding. Echoing the widely-reported "emergent abilities" of large language models when trained on increasing volume of data, we show that BASE TTS variants built with 10K+ hours and 500M+ parameters begin to demonstrate natural prosody on textually complex sentences. We design and share a specialized dataset to measure these emergent abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE TTS by evaluating against baselines that include publicly available large-scale text-to-speech systems: YourTTS, Bark and TortoiseTTS. Audio samples generated by the model can be heard at https://amazon-ltts-paper.com/.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 72 of its 105 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

72 of 105 references.

References · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  13. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  14. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  15. 2022High Fidelity Neural Audio Compressioncited by 687
  16. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  17. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  18. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  19. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  20. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  21. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  22. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  23. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  24. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  25. 2021A Survey on Neural Speech Synthesiscited by 184
  26. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  27. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  28. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  29. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  30. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  31. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  32. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  33. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 85
  34. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  35. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
  36. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  37. 2023Better speech synthesis through scalingcited by 63
  38. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 30; 2026: 11.

75 papers.

Sort:

By area:

  1. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  3. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  4. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  5. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  6. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  7. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  8. 2024Language Model Can Listen While Speakingcited by 63
  9. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  10. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  11. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  12. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
38 more
  1. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  2. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 20
  3. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  4. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  5. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  6. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 16
  7. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  8. 2024EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Controlcited by 13
  9. 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 12
  10. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
  11. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  12. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  13. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  14. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
  15. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  16. 2024Emotional Dimension Control in Language Model-Based Text-to-Speech: Spanning a Broad Spectrum of Human Emotionscited by 8
  17. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  18. 2024Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluationcited by 7
  19. 2025The NaijaVoices Dataset: Cultivating Large-Scale, High-Quality, Culturally-Rich Speech Data for African Languagescited by 7
  20. 2024Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modelingcited by 6
  21. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decodingcited by 6
  22. 2024Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesiscited by 5
  23. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  24. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusioncited by 5
  25. 2024Foundation Models for Low-Resource Language Education (Vision Paper)cited by 5
  26. 2025Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Samplingcited by 5
  27. 2024Exploring the Benefits of Tokenization of Discrete Acoustic Unitscited by 4
  28. 2024On the Effectiveness of Acoustic BPE in Decoder-Only TTScited by 4
  29. 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4
  30. 2024SF-Speech: Straightened Flow for Zero-Shot Voice Clonecited by 4
  31. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignmentcited by 4
  32. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  33. 2025The State Of TTS: A Case Study with Human Fooling Ratescited by 4
  34. 2026X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloningcited by 4
  35. 2024Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representationcited by 3
  36. 2025Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equationcited by 3
  37. 2024Enhancing Out-of-Vocabulary Performance of Indian TTS Systems for Practical Applications through Low-Effort Data Strategiescited by 2
  38. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2

50 most cited of 75 All 75 (JSON)