mlsanityData to 14 Oct 2026

SpeakStream: Streaming Text-to-Speech with Interleaved Data

arXiv 2505.19206 ↗First posted 25 May 2025cs.CL cs.LG cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS systems, typically trained and inferenced on complete utterances, introduce unacceptable delays, even with optimized inference speeds, when coupled with streaming LLM outputs. This is particularly problematic for creating responsive conversational agents where low first-token latency is critical. In this paper, we present SpeakStream, a streaming TTS system that generates audio incrementally from streaming text using a decoder-only architecture. SpeakStream is trained using a next-step prediction loss on interleaved text-speech data. During inference, it generates speech incrementally while absorbing streaming input text, making it particularly suitable for cascaded conversational AI agents where an LLM streams text to a TTS system. Our experiments demonstrate that SpeakStream achieves state-of-the-art latency results in terms of first-token latency while maintaining the quality of non-streaming TTS systems.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

27 of 36 references.

References · 27

  1. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  2. 2025Qwen2.5-Omni Technical Reportcited by 736
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  8. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  9. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  10. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  11. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  12. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
15 more
  1. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  2. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  3. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  4. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  5. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  6. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  7. 2023E3 TTS: Easy End-to-End Diffusion-based Text to Speechcited by 30
  8. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
  9. 2024LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codescited by 13
  10. 2024dMel: Speech Tokenization made Simplecited by 11
  11. 2023Speak While You Think: Streaming Speech Synthesis During Text Generationcited by 8
  12. 2023Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Predictioncited by 7
  13. 2024Zero-Shot Text-to-Speech from Continuous Text Streamscited by 7
  14. 2024Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesiscited by 6
  15. 2024Non-Causal to Causal SSL-Supported Transfer Learning: Towards a High-Performance Low-Latency Speech Vocodercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025VoXtream: Full-Stream Text-to-Speech with Extremely Low Latencycited by 5
  2. 2025ChipChat: Low-Latency Cascaded Conversational Agent in MLXcited by 4
  3. 2026CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignmentcited by 3
  4. 2026VoXtream2: Full-stream TTS with dynamic speaking rate controlcited by 3
  5. 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agentscited by 3
  6. 2026Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Inputcited by 1
  7. 2026Don't Pause: Streaming Video-Language Synchrony for Online Video Understandingcited by 1
  8. 2026X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritancecited by 0