mlsanityData to 7 Oct 2026

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

arXiv 2403.03100 ↗First posted 5 Mar 2024eess.AS cs.AI cs.CLcited by 222

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intricately encompasses various attributes (e.g., content, prosody, timbre, and acoustic details) that pose significant challenges for generation, a natural idea is to factorize speech into individual subspaces representing different attributes and generate them individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with novel factorized diffusion models to generate natural speech in a zero-shot way. Specifically, 1) we design a neural codec with factorized vector quantization (FVQ) to disentangle speech waveform into subspaces of content, prosody, timbre, and acoustic details; 2) we propose a factorized diffusion model to generate attributes in each subspace following its corresponding prompt. With this factorization design, NaturalSpeech 3 can effectively and efficiently model intricate speech with disentangled subspaces in a divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the state-of-the-art TTS systems on quality, similarity, prosody, and intelligibility, and achieves on-par quality with human recordings. Furthermore, we achieve better performance by scaling to 1B parameters and 200K hours of training data.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 65 of its 76 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

65 of 76 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-speech Generation13 Sep
  2. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training31 Jul
  3. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models13 Jun
  4. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers18 Apr
  5. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  6. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 59

  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
38 more
  1. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  7. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  12. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  13. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  14. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  15. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  16. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  17. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  18. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  19. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  20. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  21. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  22. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  23. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  24. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  25. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  26. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  27. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  28. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  29. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  30. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 90
  31. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  32. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  33. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  34. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  35. 2022Improved Masked Image Generation with Token-Criticcited by 63
  36. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  37. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  38. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 68; 2025: 80; 2026: 74.

222 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  3. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  4. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  5. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  6. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  7. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  8. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  9. 2026Qwen3-TTS Technical Reportcited by 106
  10. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  11. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 78
  12. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
38 more
  1. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  2. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  4. 2024The VoicePrivacy 2024 Challenge Evaluation Plancited by 39
  5. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
  6. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  7. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  8. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  9. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  10. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  11. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  12. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  13. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  14. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  15. 2024DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Datasetcited by 20
  16. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  17. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  18. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 19
  19. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  20. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  21. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  22. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  23. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  24. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  25. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 15
  26. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  27. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  28. 2024TTSDS -- Text-to-Speech Distribution Scorecited by 14
  29. 2024PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generationcited by 12
  30. 2024I Can Hear You: Selective Robust Training for Deepfake Audio Detectioncited by 12
  31. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  32. 2024Cross-Domain Audio Deepfake Detection: Dataset and Analysiscited by 11
  33. 2024Where are we in audio deepfake detection? A systematic analysis over generative and detection modelscited by 11
  34. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  35. 2025CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speechcited by 11
  36. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  37. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  38. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10

50 most cited of 222 All 222 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models
  2. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
  3. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
  4. 2024Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation
  5. 2024Zero-shot Voice Conversion with Diffusion Transformers
  6. 2024FlashSpeech: Efficient Zero-Shot Speech Synthesis
  7. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speech
  8. 2023Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding