mlsanityData to 7 Oct 2026

VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention

arXiv 2102.06431 ↗First posted 12 Feb 2021cs.SD cs.CL eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

This paper proposes VARA-TTS, a non-autoregressive (non-AR) text-to-speech (TTS) model using a very deep Variational Autoencoder (VDVAE) with Residual Attention mechanism, which refines the textual-to-acoustic alignment layer-wisely. Hierarchical latent variables with different temporal resolutions from the VDVAE are used as queries for residual attention module. By leveraging the coarse global alignment from previous attention layer as an extra input, the following attention layer can produce a refined version of alignment. This amortizes the burden of learning the textual-to-acoustic alignment among multiple attention layers and outperforms the use of only a single attention layer in robustness. An utterance-level speaking speed factor is computed by a jointly-trained speaking speed predictor, which takes the mean-pooled latent variables of the coarsest layer as input, to determine number of acoustic frames at inference. Experimental results show that VARA-TTS achieves slightly inferior speech quality to an AR counterpart Tacotron 2 but an order-of-magnitude speed-up at inference; and outperforms an analogous non-AR model, BVAE-TTS, in terms of speech quality.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

36 of 45 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  9. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
24 more
  1. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  2. 2015Attention-Based Models for Speech Recognitioncited by 400
  3. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  4. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  5. 2018Understanding disentangling in β-VAEcited by 332
  6. 2016Ladder Variational Autoencoderscited by 315
  7. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  9. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  10. 2018Efficient Neural Audio Synthesiscited by 218
  11. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  12. 2018Taming VAEscited by 85
  13. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  14. 2020End-to-End Adversarial Text-to-Speechcited by 77
  15. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  16. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  17. 2017Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learningcited by 67
  18. 2020RealFormer: Transformer Likes Residual Attentioncited by 62
  19. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  20. 2020Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesiscited by 54
  21. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  22. 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
  23. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  24. 2018Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesiscited by 17

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 6; 2023: 0; 2024: 2.

12 papers.

  1. 2021A Survey on Neural Speech Synthesiscited by 182
  2. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  3. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  4. 2022DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANscited by 43
  5. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  6. 2022Style-Label-Free: Cross-Speaker Style Transfer by Quantized VAE and Speaker-wise Normalization in Speech Synthesiscited by 7
  7. 2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
  8. 2022Back-Translation-Style Data Augmentation for Mandarin Chinese Polyphone Disambiguationcited by 6
  9. 2021Conditional Deep Hierarchical Variational Autoencoder for Voice Conversioncited by 2
  10. 2022Features Fusion Framework for Multimodal Irregular Time-series Eventscited by 2
  11. 2021UniTTS: Residual Learning of Unified Embedding Space for Speech Style Controlcited by 1
  12. 2022Towards zero-shot Text-based voice editing using acoustic context conditioning, utterance embeddings, and reference encoderscited by 0