# Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark

arXiv: https://arxiv.org/abs/2502.04976 · HTML: https://mlsanity.com/paper/2502.04976/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2025.

## Built on by (most cited first, 18 of 18)

- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [A Multi-Agent Framework with Structured Reasoning and Reflective Refinement for Multimodal Empathetic Response Generation](https://mlsanity.com/paper/2604.18988/) (2026)
- [PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning](https://mlsanity.com/paper/2508.09521/) (2025)
- [E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model](https://mlsanity.com/paper/2508.12854/) (2025)
- [AEQ-Bench: Measuring Empathy of Omni-Modal Large Models](https://mlsanity.com/paper/2601.10513/) (2026)
- [InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos](https://mlsanity.com/paper/2608.01157/) (2026)
- [Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy](https://mlsanity.com/paper/2603.02123/) (2026)
- [ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation](https://mlsanity.com/paper/2601.16225/) (2026)
- [MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents](https://mlsanity.com/paper/2602.21941/) (2026)
- [PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue](https://mlsanity.com/paper/2606.12902/) (2026)
- [PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation](https://mlsanity.com/paper/2606.23124/) (2026)
- [Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2607.24430/) (2026)
- [EmpaAva: An Open-source Agentic 3D-Avatar Empathetic Live Chatbot](https://mlsanity.com/paper/2608.04709/) (2026)
- [MOCC-R1: Reinforcing Reasoning-Response Consistency for Multimodal Counselor Response Generation](https://mlsanity.com/paper/2609.17180/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)
- [EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance](https://mlsanity.com/paper/2609.38182/) (2026)
- [Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue](https://mlsanity.com/paper/2609.39072/) (2026)
- [FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation](https://mlsanity.com/paper/2610.20051/) (2026)

## Abstract (arXiv)

Empathetic Response Generation (ERG) is one of the key tasks of the affective computing area, which aims to produce emotionally nuanced and compassionate responses to user's queries. However, existing ERG research is predominantly confined to the singleton text modality, limiting its effectiveness since human emotions are inherently conveyed through multiple modalities. To combat this, we introduce an avatar-based Multimodal ERG (MERG) task, entailing rich text, speech, and facial vision information. We first present a large-scale high-quality benchmark dataset, AvaMERG, which extends traditional text ERG by incorporating authentic human speech audio and dynamic talking-face avatar videos, encompassing a diverse range of avatar profiles and broadly covering various topics of real-world scenarios. Further, we deliberately tailor a system, named Empatheia, for MERG. Built upon a Multimodal Large Language Model (MLLM) with multimodal encoder, speech and avatar generators, Empatheia performs end-to-end MERG, with Chain-of-Empathetic reasoning mechanism integrated for enhanced empathy understanding and reasoning. Finally, we devise a list of empathetic-enhanced tuning strategies, strengthening the capabilities of emotional accuracy and content, avatar-profile consistency across modalities. Experimental results on AvaMERG data demonstrate that Empatheia consistently shows superior performance than baseline methods on both textual ERG and MERG. Overall, this work is expected to pioneer the MERG research by introducing a novel benchmark and an end-to-end model, laying a solid foundation for future advancements in multimodal empathetic response generation.