# Neural Speech Synthesis with Transformer Network

arXiv: https://arxiv.org/abs/1809.08895 · HTML: https://mlsanity.com/paper/1809.08895/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2018.

## Built on by (most cited first, 14 of 14)

- [Denoising Vision Transformers](https://mlsanity.com/paper/2401.02957/) (2024)
- [Speech Synthesis with Mixed Emotions](https://mlsanity.com/paper/2208.05890/) (2022)
- [ESPnet2-TTS: Extending the Edge of TTS Research](https://mlsanity.com/paper/2110.07840/) (2021)
- [Towards Natural and Controllable Cross-Lingual Voice Conversion Based on Neural TTS Model and Phonetic Posteriorgram](https://mlsanity.com/paper/2102.01991/) (2021)
- [A multilingual training strategy for low resource Text to Speech](https://mlsanity.com/paper/2409.01217/) (2024)
- [Next Tokens Denoising for Speech Synthesis](https://mlsanity.com/paper/2507.22746/) (2025)
- [Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language](https://mlsanity.com/paper/2212.08321/) (2022)
- [Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder](https://mlsanity.com/paper/2212.08329/) (2022)
- [Disambiguation of Chinese Polyphones in an End-to-End Framework with Semantic Features Extracted by Pre-trained BERT](https://mlsanity.com/paper/2501.01102/) (2025)
- [Applying Syntaxunicodex2013Prosody Mapping Hypothesis and Prosodic Well-Formedness Constraints to Neural Sequence-to-Sequence Speech Synthesis](https://mlsanity.com/paper/2203.15276/) (2022)
- [Latent Optimal Paths by Gumbel Propagation for Variational Bayesian Dynamic Programming](https://mlsanity.com/paper/2306.02568/) (2023)
- [Automatic design optimization of preference-based subjective evaluation with online learning in crowdsourcing environment](https://mlsanity.com/paper/2403.06100/) (2024)
- [Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement](https://mlsanity.com/paper/2510.01722/) (2025)
- [Investigating Group Relative Policy Optimization for Diffusion Transformer based Text-to-Audio Generation](https://mlsanity.com/paper/2603.01565/) (2026)

## Abstract (arXiv)

Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-the-art performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs). Inspired by the success of Transformer network in neural machine translation (NMT), in this paper, we introduce and adapt the multi-head attention mechanism to replace the RNN structures and also the original attention mechanism in Tacotron2. With the help of multi-head self-attention, the hidden states in the encoder and decoder are constructed in parallel, which improves the training efficiency. Meanwhile, any two inputs at different times are connected directly by self-attention mechanism, which solves the long range dependency problem effectively. Using phoneme sequences as input, our Transformer TTS network generates mel spectrograms, followed by a WaveNet vocoder to output the final audio results. Experiments are conducted to test the efficiency and performance of our new network. For the efficiency, our Transformer TTS network can speed up the training about 4.25 times faster compared with Tacotron2. For the performance, rigorous human tests show that our proposed model achieves state-of-the-art performance (outperforms Tacotron2 with a gap of 0.048) and is very close to human quality (4.39 vs 4.44 in MOS).