VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 20 of its 23 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
20 of 23 references.
References · 20
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
- 2025Qwen3-Omni Technical Reportcited by 497
- 2025Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Modelscited by 219
- 2026Qwen3-TTS Technical Reportcited by 118
- 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
- 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 63
- 2021TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global contextcited by 62
- 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
- 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
- 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 58
- 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 54
8 more
- 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 34
- 2021Lhotse: a speech data representation library for the modern deep learning ecosystemcited by 33
- 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
- 2025HiFiTTS-2: A Large-Scale High Bandwidth Speech Datasetcited by 10
- 2025SpeakStream: Streaming Text-to-Speech with Interleaved Datacited by 8
- 2024Efficient Generative Modeling with Residual Vector Quantization-Based Tokenscited by 7
- 2025DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilitiescited by 5
- 2025VoXtream: Full-Stream Text-to-Speech with Extremely Low Latencycited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
- 2023Towards human-like spoken dialogue generation between AI agents from written dialogue
- 2026FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
- 2025ChipChat: Low-Latency Cascaded Conversational Agent in MLX
- 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
- 2025PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction
- 2026Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
- 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents