mlsanityData to 14 Oct 2026

Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

arXiv 2506.00722 ↗First posted 31 May 2025cs.CL cs.SD eess.AScited by 5

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Unlike traditional cascaded pipelines, end-to-end (E2E) spoken dialogue systems preserve full differentiability and capture non-phonemic information, making them well-suited for modeling spoken interactions. However, existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence. We propose a simple yet effective strategy leveraging a chain-of-thought (CoT) formulation, ensuring that training on conversational data remains closely aligned with the multimodal language model (LM)'s pre-training on speech recognition~(ASR), text-to-speech synthesis (TTS), and text LM tasks. Our method achieves over 1.5 ROUGE-1 improvement over the baseline, successfully training spoken dialogue systems on publicly available human-human conversation datasets, while being compute-efficient enough to train on just 300 hours of public human-human conversation data, such as the Switchboard. We will publicly release our models and training code.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 48 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

30 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming29 Aug

Other references · 29

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  5. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  9. 2023Simple and Controllable Music Generationcited by 488
  10. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  11. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  12. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
17 more
  1. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  2. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  3. 2022Generative Spoken Dialogue Language Modelingcited by 126
  4. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  5. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  6. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 72
  7. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  8. 2024OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversationcited by 60
  9. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  10. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
  11. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  12. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  13. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 37
  14. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
  15. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 26
  16. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  17. 2025ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systemscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 19
  2. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usagecited by 15
  3. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 5
  4. 2026Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systemscited by 1
  5. 2026Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM
  2. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
  3. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings
  4. 2026COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
  5. 2026Learning to Simulate Human Dialogue
  6. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation
  7. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
  8. 2025DialoSpeech: Dual-Speaker Dialogue Generation with LLM and Flow Matching