mlsanityData to 7 Oct 2026

NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality

arXiv 2205.04421 ↗First posted 9 May 2022eess.AS cs.AI cs.CLcited by 151

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text to speech (TTS) has made rapid progress in both academia and industry in recent years. Some questions naturally arise that whether a TTS system can achieve human-level quality, how to define/judge that quality and how to achieve it. In this paper, we answer these questions by first defining the human-level quality based on the statistical significance of subjective measure and introducing appropriate guidelines to judge it, and then developing a TTS system called NaturalSpeech that achieves human-level quality on a benchmark dataset. Specifically, we leverage a variational autoencoder (VAE) for end-to-end text to waveform generation, with several key modules to enhance the capacity of the prior from text and reduce the complexity of the posterior from speech, including phoneme pre-training, differentiable duration modeling, bidirectional prior/posterior modeling, and a memory mechanism in VAE. Experiment evaluations on popular LJSpeech dataset show that our proposed NaturalSpeech achieves -0.01 CMOS (comparative mean opinion score) to human recordings at the sentence level, with Wilcoxon signed rank test at p-level p >> 0.05, which demonstrates no statistically significant difference from human recordings for the first time on this dataset.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 9 May 2022.

Builds on

46 of 63 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2015Variational Inference with Normalizing Flowscited by 2,161
  8. 2016Density estimation using Real NVPcited by 2,029
  9. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
34 more
  1. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  2. 2016Least Squares Generative Adversarial Networkscited by 1,000
  3. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  6. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  7. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  10. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  11. 2018SDR - half-baked or well done?cited by 420
  12. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  13. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  14. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 265
  15. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  16. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  17. 2018Efficient Neural Audio Synthesiscited by 218
  18. 2021A Survey on Neural Speech Synthesiscited by 182
  19. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  20. 2020Latent Video Transformercited by 106
  21. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  22. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  23. 2020End-to-End Adversarial Text-to-Speechcited by 77
  24. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  25. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  26. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  27. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  28. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  29. 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
  30. 2021MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Networkcited by 37
  31. 2021Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modelingcited by 31
  32. 2019Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversioncited by 26
  33. 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
  34. 2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 36; 2024: 50; 2025: 38; 2026: 15.

151 papers.

Sort:

By area:

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  2. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  4. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  5. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  6. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  7. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  8. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  9. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  10. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  11. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  12. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 106
38 more
  1. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  2. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 47
  3. 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesiscited by 44
  4. 2024Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modellingcited by 43
  5. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  6. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  7. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 32
  8. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  9. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  10. 2023VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Designcited by 25
  11. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  12. 2024LAFMA: A Latent Flow Matching Model for Text-to-Audio Generationcited by 24
  13. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  14. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  15. 2022Memories are One-to-Many Mapping Alleviators in Talking Face Generationcited by 22
  16. 2023Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictionscited by 22
  17. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  18. 2022VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbingcited by 20
  19. 2024StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbingcited by 20
  20. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  21. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  22. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 19
  23. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  24. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  25. 2023XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speechcited by 16
  26. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 15
  27. 2023EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speechcited by 14
  28. 2023Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesiscited by 13
  29. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  30. 2022JukeDrummer: Conditional Beat-aware Audio-domain Drum Accompaniment Generation via Transformer VQ-VAEcited by 11
  31. 2023Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Studycited by 10
  32. 2023An Efficient Temporary Deepfake Location Approach Based Embeddings for Partially Spoofed Audio Detectioncited by 10
  33. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  34. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  35. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  36. 2023HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Modelscited by 9
  37. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
  38. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9

50 most cited of 151 All 151 (JSON)