# EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance

arXiv: https://arxiv.org/abs/2609.38182 · HTML: https://mlsanity.com/paper/2609.38182/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning](https://mlsanity.com/paper/2503.05379/) (2025)
- [CEM: Commonsense-aware Empathetic Response Generation](https://mlsanity.com/paper/2109.05739/) (2021)
- [Ola: Pushing the Frontiers of Omni-Modal Language Model](https://mlsanity.com/paper/2502.04328/) (2025)
- [The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods](https://mlsanity.com/paper/1804.04262/) (2018)
- [SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training](https://mlsanity.com/paper/2412.15649/) (2024)
- [Knowledge Bridging for Empathetic Dialogue Generation](https://mlsanity.com/paper/2009.09708/) (2020)
- [MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space](https://mlsanity.com/paper/2405.16105/) (2024)
- [EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions](https://mlsanity.com/paper/2409.18042/) (2024)
- [VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning](https://mlsanity.com/paper/2509.24650/) (2025)
- [Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance](https://mlsanity.com/paper/2502.05236/) (2025)
- [Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance](https://mlsanity.com/paper/2510.10444/) (2025)
- [Towards Multimodal Emotional Support Conversation Systems](https://mlsanity.com/paper/2408.03650/) (2024)
- [From Personas to Talks: Revisiting the Impact of Personas on LLM-Synthesized Emotional Support Conversations](https://mlsanity.com/paper/2502.11451/) (2025)
- [STICKERCONV: Generating Multimodal Empathetic Responses from Scratch](https://mlsanity.com/paper/2402.01679/) (2024)
- [Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark](https://mlsanity.com/paper/2502.04976/) (2025)
- [EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot](https://mlsanity.com/paper/2406.15177/) (2024)
- [Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion](https://mlsanity.com/paper/2502.07203/) (2025)
- [Reinforcing Trustworthiness in Multimodal Emotional Support Systems](https://mlsanity.com/paper/2511.10011/) (2025)

## Abstract (arXiv)

Avatar-based multimodal empathetic response generation has emerged as a pivotal capability in human-centric systems, aiming to recognize user emotions and synthesize responses with synchronized text, audio, and talking-face video. Despite recent progress, existing methods still suffer from three critical limitations: (1) overlooking conflicting emotions across modalities, (2) lacking explicit multimodal synthesis guidance, and (3) neglecting inherent error propagation of multimodal response generation. To address these limitations, we propose EmAvatar, a novel framework for precise emotion perception and expressive response generation. It first performs deliberative multimodal emotion recognition by exposing inter-modal prediction conflicts and then initiates a multi-round QA process between a Conflict Inspector and an Evidence Collector to gather evidence for conflict resolution, leading to a robust, evidence-aware prediction. Regarding response generation, EmAvatar first synthesizes a composite script that couples the textual response with an expressive instruction. Moreover, to ensure high-quality synthesis, an iterative refinement mechanism evaluates and revises the script until it aligns with predefined criteria, serving as reliable guidance for subsequent audio and video synthesis. Extensive experiments across four tasks demonstrate that EmAvatar outperforms state-of-the-art methods. Our code will be publicly released.