# OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue

arXiv: https://arxiv.org/abs/2508.09600 · HTML: https://mlsanity.com/paper/2508.09600/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2025.

## Built on by (most cited first, 21 of 21)

- [InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue](https://mlsanity.com/paper/2510.13747/) (2025)
- [Cross-Modal Knowledge Distillation for Speech Large Language Models](https://mlsanity.com/paper/2509.14930/) (2025)
- [MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation](https://mlsanity.com/paper/2512.03034/) (2025)
- [UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models](https://mlsanity.com/paper/2510.22588/) (2025)
- [Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue](https://mlsanity.com/paper/2601.18281/) (2026)
- [Resurfacing Paralinguistic Awareness in Large Audio Language Models](https://mlsanity.com/paper/2603.11947/) (2026)
- [SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation](https://mlsanity.com/paper/2604.20842/) (2026)
- [Thinking in Depth, Speaking Directly: Recurrent Latent Reasoning for Paralinguistically Grounded Spoken Dialogue](https://mlsanity.com/paper/2609.37818/) (2026)
- [WEST: LLM based Speech Toolkit for Speech Understanding, Generation, and Interaction](https://mlsanity.com/paper/2509.19902/) (2025)
- [Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems](https://mlsanity.com/paper/2601.20230/) (2026)
- [Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction](https://mlsanity.com/paper/2602.24080/) (2026)
- [Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning](https://mlsanity.com/paper/2606.00851/) (2026)
- [From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decoding](https://mlsanity.com/paper/2606.16472/) (2026)
- [EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models](https://mlsanity.com/paper/2608.09189/) (2026)
- [ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation](https://mlsanity.com/paper/2601.16225/) (2026)
- [PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue](https://mlsanity.com/paper/2606.12902/) (2026)
- [Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning](https://mlsanity.com/paper/2607.02214/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework](https://mlsanity.com/paper/2608.28630/) (2026)
- [ER-EDF: A Psychology-Grounded Emotion Regulation Framework for Speech Empathetic Dialogue Generation in Large Audio-Language Models](https://mlsanity.com/paper/2609.15089/) (2026)
- [COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2609.22697/) (2026)

## Abstract (arXiv)

Empathy is crucial in enabling natural interactions within spoken dialogue systems, allowing machines to recognize and respond appropriately to paralinguistic cues such as age, gender, and emotion. Recent advancements in end-to-end speech language models, which unify speech understanding and generation, provide promising solutions. However, several challenges persist, including an over-reliance on large-scale dialogue datasets, insufficient extraction of paralinguistic cues vital for conveying empathy, and the lack of empathy-specific datasets and evaluation frameworks. To address these issues, we introduce OSUM-EChat, an open-source, end-to-end spoken dialogue system designed to enhance empathetic interactions, particularly in resource-limited settings. OSUM-EChat introduces two key innovations: (1) a three-stage understanding-driven spoken dialogue training strategy that extends the capabilities of a large speech understanding model to spoken dialogue tasks, and (2) a linguistic-paralinguistic dual thinking mechanism that integrates paralinguistic understanding through a chain of thought with dialogue generation, enabling the system to produce more empathetic responses. This approach reduces reliance on large-scale dialogue datasets while maintaining high-quality empathetic interactions. Additionally, we introduce the EChat-200K dataset, a rich corpus of empathetic speech-to-speech dialogues, and the EChat-eval benchmark, a comprehensive framework for evaluating the empathetic capabilities of dialogue systems. Experimental results demonstrate that OSUM-EChat outperforms end-to-end spoken dialogue models regarding empathetic responsiveness, validating its effectiveness.