mlsanityData to 7 Oct 2026

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers

arXiv 2304.09116 ↗First posted 18 Apr 2023eess.AS cs.AI cs.CLcited by 233

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, and styles (e.g., singing). Current large TTS systems usually quantize speech into discrete tokens and use language models to generate these tokens one by one, which suffer from unstable prosody, word skipping/repeating issue, and poor voice quality. In this paper, we develop NaturalSpeech 2, a TTS system that leverages a neural audio codec with residual vector quantizers to get the quantized latent vectors and uses a diffusion model to generate these latent vectors conditioned on text input. To enhance the zero-shot capability that is important to achieve diverse speech synthesis, we design a speech prompting mechanism to facilitate in-context learning in the diffusion model and the duration/pitch predictor. We scale NaturalSpeech 2 to large-scale datasets with 44K hours of speech and singing data and evaluate its voice quality on unseen speakers. NaturalSpeech 2 outperforms previous TTS systems by a large margin in terms of prosody/timbre similarity, robustness, and voice quality in a zero-shot setting, and performs novel zero-shot singing synthesis with only a speech prompt. Audio samples are available at https://speechresearch.github.io/naturalspeech2.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 42 of its 78 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

42 of 78 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  2. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speech3 Apr
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2023Consistency Modelscited by 1,778
  9. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
27 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  13. 2022AudioGen: Textually Guided Audio Generationcited by 292
  14. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  15. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  16. 2021A Survey on Neural Speech Synthesiscited by 182
  17. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  18. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  19. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  20. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  21. 2019Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversioncited by 26
  22. 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
  23. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoderscited by 15
  24. 2022RetrieverTTS: Modeling Decomposed Factors for Text-Based Speech Insertioncited by 10
  25. 2020TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguesecited by 9
  26. 2023Regeneration Learning: A Learning Paradigm for Data Generationcited by 8
  27. 2023FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Modelcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 95; 2025: 68; 2026: 46.

233 papers.

Sort:

By area:

  1. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  3. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  4. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  5. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  6. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  7. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  8. 2026Qwen3-TTS Technical Reportcited by 106
  9. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  10. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  11. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  12. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
38 more
  1. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  2. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  3. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  4. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  5. 2023TESS: Text-to-Text Self-Conditioned Simplex Diffusioncited by 63
  6. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  7. 2023Efficient Neural Music Generationcited by 50
  8. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  9. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  10. 2023CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Modelcited by 44
  11. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  12. 2024Zero-shot Voice Conversion with Diffusion Transformerscited by 39
  13. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  14. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  15. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  16. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  17. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  18. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  19. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  20. 2023Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesiscited by 29
  21. 2025VibeVoice Technical Reportcited by 28
  22. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  23. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  24. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Modelscited by 23
  25. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignmentcited by 23
  26. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesiscited by 22
  27. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  28. 2024SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Descriptioncited by 20
  29. 2024DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Datasetcited by 20
  30. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  31. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  32. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  33. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  34. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 15
  35. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  36. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matchingcited by 14
  37. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  38. 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 12

50 most cited of 233 All 233 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment
  2. 2024LatentSpeech: Latent Diffusion for Text-To-Speech Generation
  3. 2024RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis
  4. 2025Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model
  5. 2025LAPS-Diff: A Diffusion-Based Framework for Singing Voice Synthesis With Language Aware Prosody-Style Guided Learning
  6. 2024CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  8. 2024Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR