Liberating LLM Capabilities in Full-Duplex Speech Models
Not on the map. Why?
Abstract (arXiv)
Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs. Existing work improves spoken reasoning or full-duplex turn-taking, but still treats text as a hidden intermediate state or a subordinate modality rather than a first-class output channel. We propose Listen-Write-Speak (LWS), a text-first tri-channel paradigm in which a single autoregressive LLM continuously listens to user audio, writes visible free-form text as its primary output, and speaks a realtime oral response in parallel under a shared causal attention context. This behavior is implemented entirely through a Token Schema, requiring no architectural modifications, and learned via a two-stage data pipeline that synthesizes per-second cognitive annotations consistent with the revealed input timeline. Empirically, LWS demonstrates strong full-duplex interaction on Full-Duplex-Bench, reaches 4.72 on VoiceBench AlpacaEval, achieves 92.6% writing-speaking consistency, and consistently outperforms its internal ablations on URO-Bench. These results suggest that visible writing can serve as a first-class output channel for speech interaction without sacrificing realtime responsiveness. The code and dataset are available on the project page: https://royalzhang.com/project/lws-page/.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 4 May 2026.
Builds on
22 of 29 references.
References · 22
- 2025Qwen3 Technical Reportcited by 8,195
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
- 2025Qwen3-Omni Technical Reportcited by 451
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
- 2025Kimi-Audio Technical Reportcited by 210
- 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
- 2025Step-Audio 2 Technical Reportcited by 101
- 2024Language Model Can Listen While Speakingcited by 62
- 2025Step-Audio-R1 Technical Reportcited by 35
10 more
- 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 29
- 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
- 2025Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Modelscited by 20
- 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 16
- 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
- 2025Can Speech LLMs Think while Listening?cited by 15
- 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
- 2025Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Modelscited by 9
- 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
- 2025Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speechcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026LLMs are General Asynchronous Agentscited by 0
- 2026LLMs are General Asynchronous Agents28 Sep 2026