# LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes

arXiv: https://arxiv.org/abs/2406.02897 · HTML: https://mlsanity.com/paper/2406.02897/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2024.

## Built on by (most cited first, 13 of 13)

- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [SpeakStream: Streaming Text-to-Speech with Interleaved Data](https://mlsanity.com/paper/2505.19206/) (2025)
- [Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling](https://mlsanity.com/paper/2505.19669/) (2025)
- [From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training](https://mlsanity.com/paper/2509.20072/) (2025)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [Compact Neural TTS Voices for Accessibility](https://mlsanity.com/paper/2501.17332/) (2025)
- [SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation](https://mlsanity.com/paper/2509.01200/) (2025)
- [Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems](https://mlsanity.com/paper/2602.23266/) (2026)
- [MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Model](https://mlsanity.com/paper/2501.05787/) (2025)
- [BanglaFake: Constructing and Evaluating a Specialized Bengali Deepfake Audio Dataset](https://mlsanity.com/paper/2505.10885/) (2025)
- [Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Input](https://mlsanity.com/paper/2603.06444/) (2026)
- [X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritance](https://mlsanity.com/paper/2608.18661/) (2026)
- [Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes](https://mlsanity.com/paper/2609.30798/) (2026)

## Abstract (arXiv)

Prior works have demonstrated zero-shot text-to-speech by using a generative language model on audio tokens obtained via a neural audio codec. It is still challenging, however, to adapt them to low-latency scenarios. In this paper, we present LiveSpeech - a fully autoregressive language model-based approach for zero-shot text-to-speech, enabling low-latency streaming of the output audio. To allow multiple token prediction within a single decoding step, we propose (1) using adaptive codebook loss weights that consider codebook contribution in each frame and focus on hard instances, and (2) grouping codebooks and processing groups in parallel. Experiments show our proposed models achieve competitive results to state-of-the-art baselines in terms of content accuracy, speaker similarity, audio quality, and inference speed while being suitable for low-latency streaming applications.