mlsanityData to 7 Oct 2026

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

arXiv 2301.02111 ↗First posted 5 Jan 2023cs.CL cs.SD eess.AScited by 737

Not on the map. Why?

Cited by 115 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.

Key facts. Cited by 737 papers in the map (data through 7 Oct 2026). Builds on 46 of its 83 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

46 of 83 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
34 more
  1. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  8. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  9. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  13. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  14. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  15. 2019Understanding and Improving Layer Normalizationcited by 213
  16. 2021A Survey on Neural Speech Synthesiscited by 182
  17. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  18. 2018Neural Voice Cloning with a Few Samplescited by 100
  19. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  20. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  21. 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
  22. 2018Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition Systemcited by 45
  23. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  24. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  25. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  26. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Modelscited by 34
  27. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 32
  28. 2018Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesiscited by 30
  29. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  30. 2020An Open source Implementation of ITU-T Recommendation P.808 with Validationcited by 27
  31. 2019VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019cited by 25
  32. 2021Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speechcited by 22
  33. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoderscited by 15
  34. 2020TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguesecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 101; 2024: 236; 2025: 216; 2026: 184.

737 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2023Simple and Controllable Music Generationcited by 480
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  7. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  8. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  9. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  10. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  12. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
38 more
  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  2. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  5. 2025Kimi-Audio Technical Reportcited by 210
  6. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  7. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  8. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  9. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  10. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  11. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  12. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  13. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  14. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  15. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  16. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  17. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  18. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  19. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  20. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  21. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 118
  22. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  23. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  24. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  25. 2026Qwen3-TTS Technical Reportcited by 106
  26. 2025Step-Audio 2 Technical Reportcited by 101
  27. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  28. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 93
  29. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  30. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  31. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  32. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  33. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  34. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  35. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  36. 2024Addressing Representation Collapse in Vector Quantized Models with One Linear Layercited by 77
  37. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  38. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74

50 most cited of 737 All 737 (JSON)