mlsanityData to 14 Oct 2026

Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction

arXiv 2406.12707 ↗First posted 18 Jun 2024cs.CL cs.AI cs.SDcited by 13

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Large Language Model (LLM)-enhanced agents become increasingly prevalent in Human-AI communication, offering vast potential from entertainment to professional domains. However, current multi-modal dialogue systems overlook the acoustic information present in speech, which is crucial for understanding human communication nuances. This oversight can lead to misinterpretations of speakers' intentions, resulting in inconsistent or even contradictory responses within dialogues. To bridge this gap, in this paper, we propose PerceptiveAgent, an empathetic multi-modal dialogue system designed to discern deeper or more subtle meanings beyond the literal interpretations of words through the integration of speech modality perception. Employing LLMs as a cognitive core, PerceptiveAgent perceives acoustic information from input speech and generates empathetic responses based on speaking styles described in natural language. Experimental results indicate that PerceptiveAgent excels in contextual understanding by accurately discerning the speakers' true intentions in scenarios where the linguistic meaning is either contrary to or inconsistent with the speaker's true feelings, producing more nuanced and expressive spoken dialogues. Code is publicly available at: urlhttps://github.com/Haoqiu-Yan/PerceptiveAgent.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 31 of its 46 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

31 of 46 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  8. 2022Galactica: A Large Language Model for Sciencecited by 744
  9. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  10. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  12. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
19 more
  1. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  2. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  3. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  4. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  5. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  6. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  7. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  8. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  9. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  10. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  11. 2021Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causescited by 43
  12. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  13. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  14. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  15. 2023Prefix tuning for automated audio captioningcited by 27
  16. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  17. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  18. 2021Controllable Context-aware Conversational Speech Synthesiscited by 18
  19. 2023ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddingscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Agentic Large Language Models, a surveycited by 108
  2. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 25
  3. 2025Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathycited by 10
  4. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
  5. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 8
  6. 2025Speculative End-Turn Detector for Efficient Speech Chatbot Assistantcited by 6
  7. 2025MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speechcited by 6
  8. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  9. 2025E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Modelcited by 2
  10. 2026ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generationcited by 0
  11. 2026RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuancecited by 0
  12. 2026PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generationcited by 0
1 more
  1. 2026From Triggers to Emotions: A CPM-Grounded Appraisal Multi-Agent for Dynamic Emotional Evolution in Persona-Based Dialoguecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue
  2. 2025Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech
  3. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models
  4. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
  5. 2026SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
  6. 2025From Measurement to Expertise: Empathetic Expert Adapters for Context-Based Empathy in Conversational AI Agents
  7. 2025HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs
  8. 2024Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents