mlsanityData to 14 Oct 2026

PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue

arXiv 2606.12902 ↗First posted 11 Jun 2026cs.CLcited by 0

On the map: LLM reasoning → Audio-language model reasoning

also in 1 more
Abstract (arXiv)

Empathetic spoken dialogue systems require not only semantically appropriate responses but also emotionally aligned prosodic expression. However, cascade pipelines often discard acoustic cues during speech-to-text conversion, while end-to-end speech models lack interpretable control over emotion and knowledge integration. To address these challenges, we propose PRISM, a multi-agent framework for empathetic spoken dialogue that decouples speech perception, response generation, and speech synthesis into coordinated components. PRISM introduces a prosody-to-language translation mechanism to stabilize large language model reasoning and enables on-demand invocation of external knowledge tools for empathetic dialogue generation. Experimental results demonstrate that PRISM achieves consistent improvements in empathy, prosodic appropriateness, and text response generation quality across objective and subjective metrics. Our code is available at: https://github.com/Bxzfrm/PRISM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 37 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

30 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Qwen2.5-Omni Technical Report26 Mar

Other references · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
17 more
  1. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  4. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  5. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  6. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
  7. 2018Towards end-to-end spoken language understandingcited by 92
  8. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  9. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  10. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  11. 2025AnnaAgent: Dynamic Evolution Agent System with Multi-Session Memory for Realistic Seeker Simulationcited by 21
  12. 2025OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialoguecited by 21
  13. 2025Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmarkcited by 18
  14. 2024LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Languagecited by 12
  15. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 9
  16. 2025The Role of Prosody in Spoken Question Answeringcited by 6
  17. 2024TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLMcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction
  2. 2025Following the TRACE: A Structured Path to Empathetic Response Generation with Multi-Agent Models
  3. 2026Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue
  4. 2024Cause-Aware Empathetic Response Generation via Chain-of-Thought Fine-Tuning
  5. 2026PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation
  6. 2025EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems
  7. 2026STRIDE-ED: A Strategy-Grounded Stepwise Reasoning Framework for Empathetic Dialogue Systems
  8. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data