# Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis

arXiv: https://arxiv.org/abs/2011.03568 · HTML: https://mlsanity.com/paper/2011.03568/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2020.

## Built on by (most cited first, 30 of 38)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis](https://mlsanity.com/paper/2106.09660/) (2021)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [JetFormer: An Autoregressive Generative Model of Raw Images and Text](https://mlsanity.com/paper/2411.19722/) (2024)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Autoregressive Diffusion Transformer for Text-to-Speech Synthesis](https://mlsanity.com/paper/2406.05551/) (2024)
- [E3 TTS: Easy End-to-End Diffusion-based Text to Speech](https://mlsanity.com/paper/2311.00945/) (2023)
- [JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech](https://mlsanity.com/paper/2203.16852/) (2022)
- [fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit](https://mlsanity.com/paper/2109.06912/) (2021)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [WavThruVec: Latent speech representation as intermediate features for neural speech synthesis](https://mlsanity.com/paper/2203.16930/) (2022)
- [ClArTTS: An Open-Source Classical Arabic Text-to-Speech Corpus](https://mlsanity.com/paper/2303.00069/) (2023)
- [VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention](https://mlsanity.com/paper/2102.06431/) (2021)
- [DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation](https://mlsanity.com/paper/2310.01381/) (2023)
- [Assessing Evaluation Metrics for Speech-to-Speech Translation](https://mlsanity.com/paper/2110.13877/) (2021)
- [On-device neural speech synthesis](https://mlsanity.com/paper/2109.08710/) (2021)
- [FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis](https://mlsanity.com/paper/2207.03800/) (2022)
- [Improving Generalization for AI-Synthesized Voice Detection](https://mlsanity.com/paper/2412.19279/) (2024)
- [From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2110.08214/) (2021)
- [Acoustic Modeling for End-to-End Empathetic Dialogue Speech Synthesis Using Linguistic and Prosodic Contexts of Dialogue History](https://mlsanity.com/paper/2206.08039/) (2022)
- [Learning the joint distribution of two sequences using little or no paired data](https://mlsanity.com/paper/2212.03232/) (2022)
- [Prosody Analysis of Audiobooks](https://mlsanity.com/paper/2310.06930/) (2023)
- [Low-Latency Incremental Text-to-Speech Synthesis with Distilled Context Prediction Network](https://mlsanity.com/paper/2109.10724/) (2021)
- [Zero-Shot Long-Form Voice Cloning with Dynamic Convolution Attention](https://mlsanity.com/paper/2201.10375/) (2022)
- [Improving Autoregressive NLP Tasks via Modular Linearized Attention](https://mlsanity.com/paper/2304.08453/) (2023)
- [Rapid Speaker Adaptation in Low Resource Text to Speech Systems using Synthetic Data and Transfer learning](https://mlsanity.com/paper/2312.01107/) (2023)

## Abstract (arXiv)

We describe a sequence-to-sequence neural network which directly generates speech waveforms from text inputs. The architecture extends the Tacotron model by incorporating a normalizing flow into the autoregressive decoder loop. Output waveforms are modeled as a sequence of non-overlapping fixed-length blocks, each one containing hundreds of samples. The interdependencies of waveform samples within each block are modeled using the normalizing flow, enabling parallel training and synthesis. Longer-term dependencies are handled autoregressively by conditioning each flow on preceding blocks.This model can be optimized directly with maximum likelihood, with-out using intermediate, hand-designed features nor additional loss terms. Contemporary state-of-the-art text-to-speech (TTS) systems use a cascade of separately learned models: one (such as Tacotron) which generates intermediate features (such as spectrograms) from text, followed by a vocoder (such as WaveRNN) which generates waveform samples from the intermediate features. The proposed system, in contrast, does not use a fixed intermediate representation, and learns all parameters end-to-end. Experiments show that the proposed model generates speech with quality approaching a state-of-the-art neural TTS system, with significantly improved generation speed.