# Efficient Incremental Text-to-Speech on GPUs

arXiv: https://arxiv.org/abs/2211.13939 · HTML: https://mlsanity.com/paper/2211.13939/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 25 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency](https://mlsanity.com/paper/2111.09052/) (2021)
- [Unified Mandarin TTS Front-end Based on Distilled BERT Model](https://mlsanity.com/paper/2012.15404/) (2020)
- [Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework](https://mlsanity.com/paper/1911.02750/) (2019)
- [What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS](https://mlsanity.com/paper/2009.02035/) (2020)
- [Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning](https://mlsanity.com/paper/2008.03096/) (2020)
- [Alternate Endings: Improving Prosody for Incremental Neural TTS with Predicted Future Text Input](https://mlsanity.com/paper/2102.09914/) (2021)

## Built on by (most cited first, 1 of 1)

- [Incremental FastPitch: Chunk-based High Quality Text to Speech](https://mlsanity.com/paper/2401.01755/) (2024)

## Abstract (arXiv)

Incremental text-to-speech, also known as streaming TTS, has been increasingly applied to online speech applications that require ultra-low response latency to provide an optimal user experience. However, most of the existing speech synthesis pipelines deployed on GPU are still non-incremental, which uncovers limitations in high-concurrency scenarios, especially when the pipeline is built with end-to-end neural network models. To address this issue, we present a highly efficient approach to perform real-time incremental TTS on GPUs with Instant Request Pooling and Module-wise Dynamic Batching. Experimental results demonstrate that the proposed method is capable of producing high-quality speech with a first-chunk latency lower than 80ms under 100 QPS on a single NVIDIA A10 GPU and significantly outperforms the non-incremental twin in both concurrency and latency. Our work reveals the effectiveness of high-performance incremental TTS on GPUs.