mlsanityData to 7 Oct 2026

Speak While You Think: Streaming Speech Synthesis During Text Generation

arXiv 2309.11210 ↗First posted 20 Sep 2023eess.AS cs.CL cs.SDcited by 8

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Large Language Models (LLMs) demonstrate impressive capabilities, yet interaction with these models is mostly facilitated through text. Using Text-To-Speech to synthesize LLM outputs typically results in notable latency, which is impractical for fluent voice conversations. We propose LLM2Speech, an architecture to synthesize speech while text is being generated by an LLM which yields significant latency reduction. LLM2Speech mimics the predictions of a non-streaming teacher model while limiting the exposure to future context in order to enable streaming. It exploits the hidden embeddings of the LLM, a by-product of the text generation that contains informative semantic context. Experimental results show that LLM2Speech maintains the teacher's quality while reducing the latency to enable natural conversations.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 26 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

17 of 26 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2022AudioGen: Textually Guided Audio Generationcited by 292
  9. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  10. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  11. 2022ByT5 model for massively multilingual grapheme-to-phoneme conversioncited by 49
  12. 2021High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latencycited by 19
5 more
  1. 2015Highway Long Short-Term Memory RNNs for Distant Speech Recognitioncited by 17
  2. 2019Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Frameworkcited by 15
  3. 2022SoundChoice: Grapheme-to-Phoneme Models with Semantic Disambiguationcited by 14
  4. 2021Supervised and Unsupervised Approaches for Controlling Narrow Lexical Focus in Sequence-to-Sequence Speech Synthesiscited by 8
  5. 2022Transplantation of Conversational Speaking Style with Interjections in Sequence-to-Sequence Speech Synthesiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 2.

8 papers.

  1. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  2. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  3. 2025SpeakStream: Streaming Text-to-Speech with Interleaved Datacited by 8
  4. 2025VoXtream: Full-Stream Text-to-Speech with Extremely Low Latencycited by 5
  5. 2024Exploring the Benefits of Tokenization of Discrete Acoustic Unitscited by 4
  6. 2025SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translationcited by 2
  7. 2026Prosodic Boundary-Aware Streaming Generation for LLM-Based TTS with Streaming Text Inputcited by 1
  8. 2026Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
  2. 2025LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
  3. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation
  4. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS ↗
  5. 2026GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue
  6. 2023Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue
  7. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
  8. 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents