mlsanityData to 7 Oct 2026

High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency

arXiv 2111.09052 ↗First posted 17 Nov 2021cs.SD cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

This paper presents an end-to-end text-to-speech system with low latency on a CPU, suitable for real-time applications. The system is composed of an autoregressive attention-based sequence-to-sequence acoustic model and the LPCNet vocoder for waveform generation. An acoustic model architecture that adopts modules from both the Tacotron 1 and 2 models is proposed, while stability is ensured by using a recently proposed purely location-based attention mechanism, suitable for arbitrary sentence length generation. During inference, the decoder is unrolled and acoustic feature generation is performed in a streaming manner, allowing for a nearly constant latency which is independent from the sentence length. Experimental results show that the acoustic model can produce feature sequences with minimal latency about 31 times faster than real-time on a computer CPU and 6.5 times on a mobile CPU, enabling it to meet the conditions required for real-time applications on both devices. The full end-to-end system can generate almost natural quality speech, which is verified by listening tests.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

23 of 28 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  7. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  8. 2015Attention-Based Models for Speech Recognitioncited by 400
  9. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  10. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  11. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  12. 2018Efficient Neural Audio Synthesiscited by 218
11 more
  1. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  2. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  3. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
  4. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  5. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  6. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  7. 2016Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activationscited by 54
  8. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  9. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 41
  10. 2020Bunched LPCNet : Vocoder for Low-cost Neural Text-To-Speech Systemscited by 16
  11. 2019Non-Autoregressive Neural Text-to-Speechcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 2; 2024: 0; 2025: 1; 2026: 1.

19 papers.

Sort:

By area:

  1. 2022SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesiscited by 18
  2. 2023LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speechcited by 13
  3. 2021Prosodic Clustering for Phoneme-level Prosody Control in End-to-End Speech Synthesiscited by 8
  4. 2023Speak While You Think: Streaming Speech Synthesis During Text Generationcited by 8
  5. 2021Word-Level Style Control for Expressive, Non-attentive Speech Synthesiscited by 5
  6. 2025Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTScited by 5
  7. 2021Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Controlcited by 3
  8. 2022Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciationcited by 3
  9. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  10. 2022Into-TTS : Intonation Template Based Prosody Control Systemcited by 2
  11. 2022Self-supervised learning for robust voice cloningcited by 2
  12. 2022Fine-grained Noise Control for Multispeaker Speech Synthesiscited by 2
7 more
  1. 2022Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Featurescited by 2
  2. 2022Karaoker: Alignment-free singing voice synthesis with speech training datacited by 1
  3. 2022Efficient Incremental Text-to-Speech on GPUscited by 1
  4. 2022Bunched LPCNet2: Efficient Neural Vocoders Covering Devices from Cloud to Edgecited by 0
  5. 2022Generating Multilingual Gender-Ambiguous Text-to-Speech Voicescited by 0
  6. 2022Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesiscited by 0
  7. 2026Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billingcited by 0