mlsanityData to 7 Oct 2026

VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

arXiv 2406.05370 ↗First posted 8 Jun 2024cs.CL cs.SD eess.AScited by 119

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces VALL-E 2, the latest advancement in neural codec language models that marks a milestone in zero-shot text-to-speech synthesis (TTS), achieving human parity for the first time. Based on its predecessor, VALL-E, the new iteration introduces two significant enhancements: Repetition Aware Sampling refines the original nucleus sampling process by accounting for token repetition in the decoding history. It not only stabilizes the decoding but also circumvents the infinite loop issue. Grouped Code Modeling organizes codec codes into groups to effectively shorten the sequence length, which not only boosts inference speed but also addresses the challenges of long sequence modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E 2 surpasses previous systems in speech robustness, naturalness, and speaker similarity. It is the first of its kind to reach human parity on these benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech, even for sentences that are traditionally challenging due to their complexity or repetitive phrases. The advantages of this work could contribute to valuable endeavors, such as generating speech for individuals with aphasia or people with amyotrophic lateral sclerosis. See https://aka.ms/valle2 for demos of VALL-E 2.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 8 Jun 2024.

Builds on

44 of 52 references.

References · 44

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  10. 2022High Fidelity Neural Audio Compressioncited by 673
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
32 more
  1. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  2. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  4. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  5. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  6. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  7. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  8. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  9. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  10. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  11. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  12. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  13. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  14. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  15. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  16. 2018Neural Voice Cloning with a Few Samplescited by 100
  17. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  18. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  19. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  20. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
  21. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  22. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  23. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  24. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  25. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  26. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  27. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  28. 2023UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocodingcited by 37
  29. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  30. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  31. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  32. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 57; 2026: 35.

119 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  3. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  4. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  5. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  6. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  7. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  8. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  9. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  10. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  11. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  12. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
38 more
  1. 2025Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learningcited by 39
  2. 2025SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generationcited by 29
  3. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  4. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  5. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  6. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 16
  7. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  8. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  9. 2024Asynchronous Tool Usage for Real-Time Agentscited by 13
  10. 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 11
  11. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  12. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  13. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  14. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  15. 2024AccentBox: Towards High-Fidelity Zero-Shot Accent Generationcited by 9
  16. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  17. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  18. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  19. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  20. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  21. 2025MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generationcited by 7
  22. 2025Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Modelcited by 7
  23. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 7
  24. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 6
  25. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  26. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  27. 2025Step-Audio-EditX Technical Reportcited by 6
  28. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  29. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusioncited by 5
  30. 2024KALL-E:Autoregressive Speech Synthesis with Next-Distribution Predictioncited by 5
  31. 2025DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Modelcited by 5
  32. 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4
  33. 2024SF-Speech: Straightened Flow for Zero-Shot Voice Clonecited by 4
  34. 2025Voice Cloning: Comprehensive Surveycited by 4
  35. 2025OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matchingcited by 4
  36. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Ratecited by 4
  37. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  38. 2025UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoEcited by 4

50 most cited of 119 All 119 (JSON)