# Towards Multimodal Emotional Support Conversation Systems

arXiv: https://arxiv.org/abs/2408.03650 · HTML: https://mlsanity.com/paper/2408.03650/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2024.

## Built on by (most cited first, 23 of 23)

- [SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions](https://mlsanity.com/paper/2412.04036/) (2024)
- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement](https://mlsanity.com/paper/2506.03541/) (2025)
- [DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset](https://mlsanity.com/paper/2505.19978/) (2025)
- [Emotional Support with LLM-based Empathetic Dialogue Generation](https://mlsanity.com/paper/2507.12820/) (2025)
- [When LLMs Team Up: The Emergence of Collaborative Affective Computing](https://mlsanity.com/paper/2506.01698/) (2025)
- [PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning](https://mlsanity.com/paper/2508.09521/) (2025)
- [Empathetic Response in Audio-Visual Conversations Using Emotion Preference Optimization and MambaCompressor](https://mlsanity.com/paper/2412.17572/) (2024)
- [Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios](https://mlsanity.com/paper/2601.15290/) (2025)
- [Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy](https://mlsanity.com/paper/2603.02123/) (2026)
- [DMT-CBT: Longitudinal Therapeutic State Modeling for CBT Counseling](https://mlsanity.com/paper/2606.03132/) (2026)
- [E-THER: A Multimodal Dataset for Empathic AI -- Towards Emotional Mismatch Awareness](https://mlsanity.com/paper/2509.02100/) (2025)
- [Context-Emotion Aware Therapeutic Dialogue Generation: A Multi-component Reinforcement Learning Approach to Language Models for Mental Health Support](https://mlsanity.com/paper/2511.11884/) (2025)
- [DELTA: Deliberative Multi-Agent Reasoning with Reinforcement Learning for Multimodal Psychological Counseling](https://mlsanity.com/paper/2602.04112/) (2026)
- [MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents](https://mlsanity.com/paper/2602.21941/) (2026)
- [Cognitive Policy-Driven LLM for Diagnosis and Intervention of Cognitive Distortions in Emotional Support Conversation](https://mlsanity.com/paper/2604.17178/) (2026)
- [Designing Ethical Learning for Agentic AI: Toegye Yi Hwang's Ethical Emotion Regulation Framework](https://mlsanity.com/paper/2604.26958/) (2026)
- [When Clients Stop Following: A Cognitive Conceptualization Diagram-driven Framework for Strategic Counseling](https://mlsanity.com/paper/2606.04389/) (2026)
- [EmoLASP: Emotion Recognition with Language Models and Answer Set Programming](https://mlsanity.com/paper/2608.29035/) (2026)
- [MOCC-R1: Reinforcing Reasoning-Response Consistency for Multimodal Counselor Response Generation](https://mlsanity.com/paper/2609.17180/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)
- [EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance](https://mlsanity.com/paper/2609.38182/) (2026)
- [FACES: Fine-grained Gated Semantic Alignment for Multimodal Emotional Support Conversation](https://mlsanity.com/paper/2610.20051/) (2026)

## Abstract (arXiv)

The integration of conversational artificial intelligence (AI) into mental health care promises a new horizon for therapist-client interactions, aiming to closely emulate the depth and nuance of human conversations. Despite the potential, the current landscape of conversational AI is markedly limited by its reliance on single-modal data, constraining the systems' ability to empathize and provide effective emotional support. This limitation stems from a paucity of resources that encapsulate the multimodal nature of human communication essential for therapeutic counseling. To address this gap, we introduce the Multimodal Emotional Support Conversation (MESC) dataset, a first-of-its-kind resource enriched with comprehensive annotations across text, audio, and video modalities. This dataset captures the intricate interplay of user emotions, system strategies, system emotion, and system responses, setting a new precedent in the field. Leveraging the MESC dataset, we propose a general Sequential Multimodal Emotional Support framework (SMES) grounded in Therapeutic Skills Theory. Tailored for multimodal dialogue systems, the SMES framework incorporates an LLM-based reasoning model that sequentially generates user emotion recognition, system strategy prediction, system emotion prediction, and response generation. Our rigorous evaluations demonstrate that this framework significantly enhances the capability of AI systems to mimic therapist behaviors with heightened empathy and strategic responsiveness. By integrating multimodal data in this innovative manner, we bridge the critical gap between emotion recognition and emotional support, marking a significant advancement in conversational AI for mental health support.