# SpeakStream: Streaming Text-to-Speech with Interleaved Data

arXiv: https://arxiv.org/abs/2505.19206 · HTML: https://mlsanity.com/paper/2505.19206/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 27 of 27)

- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [Spirit LM: Interleaved Spoken and Written Language Model](https://mlsanity.com/paper/2402.05755/) (2024)
- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing](https://mlsanity.com/paper/2203.09690/) (2022)
- [E3 TTS: Easy End-to-End Diffusion-based Text to Speech](https://mlsanity.com/paper/2311.00945/) (2023)
- [VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech](https://mlsanity.com/paper/2401.14321/) (2024)
- [LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes](https://mlsanity.com/paper/2406.02897/) (2024)
- [dMel: Speech Tokenization made Simple](https://mlsanity.com/paper/2407.15835/) (2024)
- [Speak While You Think: Streaming Speech Synthesis During Text Generation](https://mlsanity.com/paper/2309.11210/) (2023)
- [Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction](https://mlsanity.com/paper/2311.02898/) (2023)
- [Zero-Shot Text-to-Speech from Continuous Text Streams](https://mlsanity.com/paper/2410.00767/) (2024)
- [Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis](https://mlsanity.com/paper/2412.16102/) (2024)
- [Non-Causal to Causal SSL-Supported Transfer Learning: Towards a High-Performance Low-Latency Speech Vocoder](https://mlsanity.com/paper/2408.11842/) (2024)

## Built on by (most cited first, 8 of 8)

- [VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency](https://mlsanity.com/paper/2509.15969/) (2025)
- [ChipChat: Low-Latency Cascaded Conversational Agent in MLX](https://mlsanity.com/paper/2509.00078/) (2025)
- [CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment](https://mlsanity.com/paper/2602.19574/) (2026)
- [VoXtream2: Full-stream TTS with dynamic speaking rate control](https://mlsanity.com/paper/2603.13518/) (2026)
- [VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents](https://mlsanity.com/paper/2608.13831/) (2026)
- [Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input](https://mlsanity.com/paper/2603.06444/) (2026)
- [Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding](https://mlsanity.com/paper/2606.06991/) (2026)
- [X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritance](https://mlsanity.com/paper/2608.18661/) (2026)

## Abstract (arXiv)

The latency bottleneck of traditional text-to-speech (TTS) systems fundamentally hinders the potential of streaming large language models (LLMs) in conversational AI. These TTS systems, typically trained and inferenced on complete utterances, introduce unacceptable delays, even with optimized inference speeds, when coupled with streaming LLM outputs. This is particularly problematic for creating responsive conversational agents where low first-token latency is critical. In this paper, we present SpeakStream, a streaming TTS system that generates audio incrementally from streaming text using a decoder-only architecture. SpeakStream is trained using a next-step prediction loss on interleaved text-speech data. During inference, it generates speech incrementally while absorbing streaming input text, making it particularly suitable for cascaded conversational AI agents where an LLM streams text to a TTS system. Our experiments demonstrate that SpeakStream achieves state-of-the-art latency results in terms of first-token latency while maintaining the quality of non-streaming TTS systems.