# High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency

arXiv: https://arxiv.org/abs/2111.09052 · HTML: https://mlsanity.com/paper/2111.09052/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 17 Nov 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://mlsanity.com/paper/1710.08969/) (2017)
- [Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations](https://mlsanity.com/paper/1606.01305/) (2016)
- [Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis](https://mlsanity.com/paper/1910.10288/) (2019)
- [MelNet: A Generative Model for Audio in the Frequency Domain](https://mlsanity.com/paper/1906.01083/) (2019)
- [Bunched LPCNet : Vocoder for Low-cost Neural Text-To-Speech Systems](https://mlsanity.com/paper/2008.04574/) (2020)
- [Non-Autoregressive Neural Text-to-Speech](https://mlsanity.com/paper/1905.08459/) (2019)

## Built on by (most cited first, 19 of 19)

- [SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis](https://mlsanity.com/paper/2204.03040/) (2022)
- [LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2308.16569/) (2023)
- [Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesis](https://mlsanity.com/paper/2111.10177/) (2021)
- [Speak While You Think: Streaming Speech Synthesis During Text Generation](https://mlsanity.com/paper/2309.11210/) (2023)
- [Word-Level Style Control for Expressive, Non-attentive Speech Synthesis](https://mlsanity.com/paper/2111.10173/) (2021)
- [Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTS](https://mlsanity.com/paper/2508.04721/) (2025)
- [Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control](https://mlsanity.com/paper/2111.10168/) (2021)
- [Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation](https://mlsanity.com/paper/2210.17264/) (2022)
- [Controllable speech synthesis by learning discrete phoneme-level prosodic representations](https://mlsanity.com/paper/2211.16307/) (2022)
- [Into-TTS : Intonation Template Based Prosody Control System](https://mlsanity.com/paper/2204.01271/) (2022)
- [Self-supervised learning for robust voice cloning](https://mlsanity.com/paper/2204.03421/) (2022)
- [Fine-grained Noise Control for Multispeaker Speech Synthesis](https://mlsanity.com/paper/2204.05070/) (2022)
- [Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features](https://mlsanity.com/paper/2211.00342/) (2022)
- [Karaoker: Alignment-free singing voice synthesis with speech training data](https://mlsanity.com/paper/2204.04127/) (2022)
- [Efficient Incremental Text-to-Speech on GPUs](https://mlsanity.com/paper/2211.13939/) (2022)
- [Bunched LPCNet2: Efficient Neural Vocoders Covering Devices from Cloud to Edge](https://mlsanity.com/paper/2203.14416/) (2022)
- [Generating Multilingual Gender-Ambiguous Text-to-Speech Voices](https://mlsanity.com/paper/2211.00375/) (2022)
- [Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis](https://mlsanity.com/paper/2211.01327/) (2022)
- [Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing](https://mlsanity.com/paper/2610.00063/) (2026)

## Abstract (arXiv)

This paper presents an end-to-end text-to-speech system with low latency on a CPU, suitable for real-time applications. The system is composed of an autoregressive attention-based sequence-to-sequence acoustic model and the LPCNet vocoder for waveform generation. An acoustic model architecture that adopts modules from both the Tacotron 1 and 2 models is proposed, while stability is ensured by using a recently proposed purely location-based attention mechanism, suitable for arbitrary sentence length generation. During inference, the decoder is unrolled and acoustic feature generation is performed in a streaming manner, allowing for a nearly constant latency which is independent from the sentence length. Experimental results show that the acoustic model can produce feature sequences with minimal latency about 31 times faster than real-time on a computer CPU and 6.5 times on a mobile CPU, enabling it to meet the conditions required for real-time applications on both devices. The full end-to-end system can generate almost natural quality speech, which is verified by listening tests.