# Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems

arXiv: https://arxiv.org/abs/2602.23266 · HTML: https://mlsanity.com/paper/2602.23266/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 26 of 26)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [LLaMA-Omni: Seamless Speech Interaction with Large Language Models](https://mlsanity.com/paper/2409.06666/) (2024)
- [FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs](https://mlsanity.com/paper/2407.04051/) (2024)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words](https://mlsanity.com/paper/2406.13340/) (2024)
- [Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation](https://mlsanity.com/paper/2406.07867/) (2024)
- [Mining Discourse Markers for Unsupervised Sentence Representation Learning](https://mlsanity.com/paper/1903.11850/) (2019)
- [FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations](https://mlsanity.com/paper/2509.06502/) (2025)
- [Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention](https://mlsanity.com/paper/1609.06380/) (2016)
- [Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework](https://mlsanity.com/paper/1911.02750/) (2019)
- [LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes](https://mlsanity.com/paper/2406.02897/) (2024)
- [Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency](https://mlsanity.com/paper/2106.02228/) (2021)
- [Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models](https://mlsanity.com/paper/2410.02205/) (2024)
- [X-Talk: On the Underestimated Potential of Modular Speech-to-Speech Dialogue System](https://mlsanity.com/paper/2512.18706/) (2025)
- [SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs](https://mlsanity.com/paper/2505.19163/) (2025)
- [ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems](https://mlsanity.com/paper/2503.08533/) (2025)
- [Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference](https://mlsanity.com/paper/1907.09692/) (2019)

## Built on by (most cited first, 2 of 2)

- [RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue](https://mlsanity.com/paper/2603.23346/) (2026)
- [Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models](https://mlsanity.com/paper/2607.19932/) (2026)

## Abstract (arXiv)

Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.