# EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot

arXiv: https://arxiv.org/abs/2406.15177 · HTML: https://mlsanity.com/paper/2406.15177/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2024.

## Built on by (most cited first, 17 of 17)

- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [Personalized Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2412.02142/) (2024)
- [PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2408.09481/) (2024)
- [SPHERE: An Evaluation Card for Human-AI Systems](https://mlsanity.com/paper/2504.07971/) (2025)
- [Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers](https://mlsanity.com/paper/2510.04577/) (2025)
- [AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues](https://mlsanity.com/paper/2412.17292/) (2024)
- [E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model](https://mlsanity.com/paper/2508.12854/) (2025)
- [Empathetic Response in Audio-Visual Conversations Using Emotion Preference Optimization and MambaCompressor](https://mlsanity.com/paper/2412.17572/) (2024)
- [A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents](https://mlsanity.com/paper/2601.10809/) (2026)
- [DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution](https://mlsanity.com/paper/2603.19248/) (2026)
- [Generative Emotion Cause Explanation in Multimodal Conversations](https://mlsanity.com/paper/2411.02430/) (2024)
- [Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey](https://mlsanity.com/paper/2604.11283/) (2026)
- [PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation](https://mlsanity.com/paper/2606.23124/) (2026)
- [Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2607.24430/) (2026)
- [EmpaAva: An Open-source Agentic 3D-Avatar Empathetic Live Chatbot](https://mlsanity.com/paper/2608.04709/) (2026)
- [MOCC-R1: Reinforcing Reasoning-Response Consistency for Multimodal Counselor Response Generation](https://mlsanity.com/paper/2609.17180/) (2026)
- [EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance](https://mlsanity.com/paper/2609.38182/) (2026)

## Abstract (arXiv)

This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access.