mlsanityData to 14 Oct 2026

Review of end-to-end speech synthesis technology based on deep learning

arXiv 2104.09995 ↗First posted 20 Apr 2021cs.SD cs.CL eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

As an indispensable part of modern human-computer interaction system, speech synthesis technology helps users get the output of intelligent machine more easily and intuitively, thus has attracted more and more attention. Due to the limitations of high complexity and low efficiency of traditional speech synthesis technology, the current research focus is the deep learning-based end-to-end speech synthesis technology, which has more powerful modeling ability and a simpler pipeline. It mainly consists of three modules: text front-end, acoustic model, and vocoder. This paper reviews the research status of these three parts, and classifies and compares various methods according to their emphasis. Moreover, this paper also summarizes the open-source speech corpus of English, Chinese and other languages that can be used for speech synthesis tasks, and introduces some commonly used subjective and objective speech quality evaluation method. Finally, some attractive future research directions are pointed out.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 127 of its 259 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

127 of 259 references.

References · 127

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2013Auto-Encoding Variational Bayescited by 4,053
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  11. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  12. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
38 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  2. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  3. 2018Demystifying MMD GANscited by 1,293
  4. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  8. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  9. 2016Pixel Recurrent Neural Networkscited by 816
  10. 2018Self-Attention Generative Adversarial Networkscited by 809
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  13. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  14. 2016Language Modeling with Gated Convolutional Networkscited by 702
  15. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  16. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  17. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  18. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  19. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  20. 2015A Neural Algorithm of Artistic Stylecited by 483
  21. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  22. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  23. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  24. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  25. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  26. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  27. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  28. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  29. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  30. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  31. 2016Variational Lossy Autoencodercited by 176
  32. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  33. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  34. 2019An Attentive Survey of Attention Modelscited by 124
  35. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  36. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  37. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 110
  38. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 3.

6 papers.

  1. 2021A Survey on Neural Speech Synthesiscited by 184
  2. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
  3. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  4. 2023Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-artcited by 14
  5. 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
  6. 2024Audio-based Kinship Verification Using Age Domain Conversioncited by 3