mlsanityData to 14 Oct 2026

Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech

arXiv 2106.12896 ↗First posted 24 Jun 2021cs.SD cs.AI cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Whilst recent neural text-to-speech (TTS) approaches produce high-quality speech, they typically require a large amount of recordings from the target speaker. In previous work, a 3-step method was proposed to generate high-quality TTS while greatly reducing the amount of data required for training. However, we have observed a ceiling effect in the level of naturalness achievable for highly expressive voices when using this approach. In this paper, we present a method for building highly expressive TTS voices with as little as 15 minutes of speech data from the target speaker. Compared to the current state-of-the-art approach, our proposed improvements close the gap to recordings by 23.3% for naturalness of speech and by 16.3% for speaker similarity. Further, we match the naturalness and speaker similarity of a Tacotron2-based full-data (~10 hours) model using only 15 minutes of target speaker data, whereas with 30 minutes or more, we significantly outperform it. The following improvements are proposed: 1) changing from an autoregressive, attention-based TTS model to a non-autoregressive model replacing attention with an external duration model and 2) an additional Conditional Generative Adversarial Network (cGAN) based fine-tuning step.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2014Conditional Generative Adversarial Netscited by 2,162
  5. 2018Self-Attention Generative Adversarial Networkscited by 809
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  8. 2015Generating Sentences from a Continuous Spacecited by 612
  9. 2017Generalized End-to-End Loss for Speaker Verificationcited by 293
  10. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  11. 2018Efficient Neural Audio Synthesiscited by 219
  12. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
20 more
  1. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  2. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  3. 2018The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methodscited by 68
  4. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  5. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  6. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 46
  7. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  8. 2020LRSpeech: Extremely Low-Resource Speech Synthesis and Recognitioncited by 43
  9. 2017Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networkscited by 38
  10. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  11. 2018Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesiscited by 30
  12. 2019End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learningcited by 30
  13. 2020Low-resource expressive text-to-speech using data augmentationcited by 26
  14. 2021Universal Neural Vocoding with Parallel WaveNetcited by 25
  15. 2020CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speechcited by 22
  16. 2019Exploring Transfer Learning for Low Resource Emotional TTScited by 21
  17. 2018Effect of data reduction on sequence-to-sequence neural TTScited by 16
  18. 2020Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languagescited by 16
  19. 2020AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIANcited by 14
  20. 2019A New GAN-based End-to-End TTS Training Algorithmcited by 13

Built on by

Cited by year

Citing papers per year: 2022: 8; 2023: 5; 2024: 0; 2025: 1.

14 papers.

Sort:

By area:

  1. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  2. 2022Cross-speaker style transfer for text-to-speech using data augmentationcited by 9
  3. 2022Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing modulecited by 8
  4. 2022Cross-Speaker Emotion Transfer for Low-Resource Text-to-Speech Using Non-Parallel Voice Conversion with Pitch-Shift Data Augmentationcited by 8
  5. 2022Distribution augmentation for low-resource expressive text-to-speechcited by 5
  6. 2022Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)cited by 5
  7. 2022Computer-assisted Pronunciation Training -- Speech synthesis is almost all you needcited by 4
  8. 2023Modelling low-resource accents without accent-specific TTS frontendcited by 3
  9. 2022GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversioncited by 2
  10. 2023Comparing normalizing flows and diffusion models for prosody and acoustic modelling in text-to-speechcited by 2
  11. 2023AE-Flow: AutoEncoder Normalizing Flowcited by 2
  12. 2023Creating New Voices using Normalizing Flowscited by 1
2 more
  1. 2023Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speechcited by 0
  2. 2025Dhvani: A Weakly-supervised Phonemic Error Detection and Personalized Feedback System for Hindicited by 0