mlsanityData to 14 Oct 2026

Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead

arXiv 2606.21882 ↗First posted 20 Jun 2026cs.SD cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 54 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

38 of 54 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  12. 2025Qwen2.5-Omni Technical Reportcited by 736
26 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  3. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  4. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  6. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  8. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  9. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  10. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  11. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  12. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  13. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  14. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  15. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  16. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  17. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  18. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  19. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  20. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 58
  21. 2025Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptationcited by 28
  22. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
  23. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 17
  24. 2019Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Frameworkcited by 15
  25. 2020What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTScited by 9
  26. 2024Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speechcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUscited by 0