# Non-Autoregressive TTS with Explicit Duration Modelling for Low-Resource Highly Expressive Speech

arXiv: https://arxiv.org/abs/2106.12896 · HTML: https://mlsanity.com/paper/2106.12896/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods](https://mlsanity.com/paper/1804.04262/) (2018)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion](https://mlsanity.com/paper/1904.04631/) (2019)
- [Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis](https://mlsanity.com/paper/1910.10288/) (2019)
- [LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition](https://mlsanity.com/paper/2008.03687/) (2020)
- [Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks](https://mlsanity.com/paper/1704.00849/) (2017)
- [Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS](https://mlsanity.com/paper/1906.00672/) (2019)
- [Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis](https://mlsanity.com/paper/1808.10128/) (2018)
- [End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning](https://mlsanity.com/paper/1904.06508/) (2019)
- [Low-resource expressive text-to-speech using data augmentation](https://mlsanity.com/paper/2011.05707/) (2020)
- [Universal Neural Vocoding with Parallel WaveNet](https://mlsanity.com/paper/2102.01106/) (2021)
- [CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech](https://mlsanity.com/paper/2004.14617/) (2020)
- [Exploring Transfer Learning for Low Resource Emotional TTS](https://mlsanity.com/paper/1901.04276/) (2019)
- [Effect of data reduction on sequence-to-sequence neural TTS](https://mlsanity.com/paper/1811.06315/) (2018)
- [Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages](https://mlsanity.com/paper/2008.04549/) (2020)

## Built on by (most cited first, 14 of 14)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Cross-speaker style transfer for text-to-speech using data augmentation](https://mlsanity.com/paper/2202.05083/) (2022)
- [Voice Filter: Few-shot text-to-speech speaker adaptation using voice conversion as a post-processing module](https://mlsanity.com/paper/2202.08164/) (2022)
- [Cross-Speaker Emotion Transfer for Low-Resource Text-to-Speech Using Non-Parallel Voice Conversion with Pitch-Shift Data Augmentation](https://mlsanity.com/paper/2204.10020/) (2022)
- [Distribution augmentation for low-resource expressive text-to-speech](https://mlsanity.com/paper/2202.06409/) (2022)
- [Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)](https://mlsanity.com/paper/2207.01547/) (2022)
- [Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need](https://mlsanity.com/paper/2207.00774/) (2022)
- [Modelling low-resource accents without accent-specific TTS frontend](https://mlsanity.com/paper/2301.04606/) (2023)
- [GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversion](https://mlsanity.com/paper/2207.01454/) (2022)
- [Comparing normalizing flows and diffusion models for prosody and acoustic modelling in text-to-speech](https://mlsanity.com/paper/2307.16679/) (2023)
- [AE-Flow: AutoEncoder Normalizing Flow](https://mlsanity.com/paper/2312.16552/) (2023)
- [Creating New Voices using Normalizing Flows](https://mlsanity.com/paper/2312.14569/) (2023)
- [Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech](https://mlsanity.com/paper/2309.08255/) (2023)
- [Dhvani: A Weakly-supervised Phonemic Error Detection and Personalized Feedback System for Hindi](https://mlsanity.com/paper/2506.02166/) (2025)

## Abstract (arXiv)

Whilst recent neural text-to-speech (TTS) approaches produce high-quality speech, they typically require a large amount of recordings from the target speaker. In previous work, a 3-step method was proposed to generate high-quality TTS while greatly reducing the amount of data required for training. However, we have observed a ceiling effect in the level of naturalness achievable for highly expressive voices when using this approach. In this paper, we present a method for building highly expressive TTS voices with as little as 15 minutes of speech data from the target speaker. Compared to the current state-of-the-art approach, our proposed improvements close the gap to recordings by 23.3% for naturalness of speech and by 16.3% for speaker similarity. Further, we match the naturalness and speaker similarity of a Tacotron2-based full-data (~10 hours) model using only 15 minutes of target speaker data, whereas with 30 minutes or more, we significantly outperform it. The following improvements are proposed: 1) changing from an autoregressive, attention-based TTS model to a non-autoregressive model replacing attention with an external duration model and 2) an additional Conditional Generative Adversarial Network (cGAN) based fine-tuning step.