E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model
Not on the map. Why?
Abstract (arXiv)
Multimodal Empathetic Response Generation (MERG) is crucial for building emotionally intelligent human-computer interactions. Although large language models (LLMs) have improved text-based ERG, challenges remain in handling multimodal emotional content and maintaining identity consistency. Thus, we propose E3RG, an Explicit Emotion-driven Empathetic Response Generation System based on multimodal LLMs which decomposes MERG task into three parts: multimodal empathy understanding, empathy memory retrieval, and multimodal response generation. By integrating advanced expressive speech and video generative models, E3RG delivers natural, emotionally rich, and identity-consistent responses without extra training. Experiments validate the superiority of our system on both zero-shot and few-shot settings, securing Top-1 position in the Avatar-based Multimodal Empathy Challenge on ACM MM 25. Our code is available at https://github.com/RH-Lin/E3RG.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 18 Aug 2025.
Builds on
38 of 66 references.
References · 38
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
- 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
- 2021Towards Emotional Support Dialog Systemscited by 299
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
- 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
26 more
- 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
- 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
- 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
- 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
- 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
- 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
- 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
- 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
- 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
- 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
- 2021Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesiscited by 56
- 2023Is ChatGPT Equipped with Emotional Dialogue Capabilities?cited by 54
- 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
- 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
- 2024LLM Voting: Human Choices and AI Collective Decision Makingcited by 48
- 2023SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Supportcited by 46
- 2023Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvementscited by 42
- 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
- 2024STICKERCONV: Generating Multimodal Empathetic Responses from Scratchcited by 22
- 2025Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmarkcited by 18
- 2024EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbotcited by 17
- 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
- 2024Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligencecited by 11
- 2025Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generationcited by 11
- 2023E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generationcited by 4
- 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimization
- 2026EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
- 2025EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment
- 2024Cause-Aware Empathetic Response Generation via Chain-of-Thought Fine-Tuning
- 2026E3mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
- 2023DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
- 2024TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM
- 2026PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation