# E-THER: A Multimodal Dataset for Empathic AI -- Towards Emotional Mismatch Awareness

arXiv: https://arxiv.org/abs/2509.02100 · HTML: https://mlsanity.com/paper/2509.02100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 72 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [MoEL: Mixture of Empathetic Listeners](https://mlsanity.com/paper/1908.07687/) (2019)
- [Modeling Empathy and Distress in Reaction to News Stories](https://mlsanity.com/paper/1808.10399/) (2018)
- [Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements](https://mlsanity.com/paper/2310.05140/) (2023)
- [MODMA dataset: a Multi-modal Open Dataset for Mental-disorder Analysis](https://mlsanity.com/paper/2002.09283/) (2020)
- [Towards Multimodal Emotional Support Conversation Systems](https://mlsanity.com/paper/2408.03650/) (2024)
- [STICKERCONV: Generating Multimodal Empathetic Responses from Scratch](https://mlsanity.com/paper/2402.01679/) (2024)
- [Are Large Language Models More Empathetic than Humans?](https://mlsanity.com/paper/2406.05063/) (2024)
- [MEDIC: A Multimodal Empathy Dataset in Counseling](https://mlsanity.com/paper/2305.02842/) (2023)
- [Multi-dimensional Evaluation of Empathetic Dialog Responses](https://mlsanity.com/paper/2402.11409/) (2024)
- [Contextual Emotion Recognition using Large Vision Language Models](https://mlsanity.com/paper/2405.08992/) (2024)
- [Psychological Metrics for Dialog System Evaluation](https://mlsanity.com/paper/2305.14757/) (2023)

## Abstract (arXiv)

A prevalent shortfall among current empathic AI systems is their inability to recognize when verbal expressions may not fully reflect underlying emotional states. This is because the existing datasets, used for the training of these systems, focus on surface-level emotion recognition without addressing the complex verbal-visual incongruence (mismatch) patterns useful for empathic understanding. In this paper, we present E-THER, the first Person-Centered Therapy-grounded multimodal dataset with multidimensional annotations for verbal-visual incongruence detection, enabling training of AI systems that develop genuine rather than performative empathic capabilities. The annotations included in the dataset are drawn from humanistic approach, i.e., identifying verbal-visual emotional misalignment in client-counsellor interactions - forming a framework for training and evaluating AI on empathy tasks. Additional engagement scores provide behavioral annotations for research applications. Notable gains in empathic and therapeutic conversational qualities are observed in state-of-the-art vision-language models (VLMs), such as IDEFICS and VideoLLAVA, using evaluation metrics grounded in empathic and therapeutic principles. Empirical findings indicate that our incongruence-trained models outperform general-purpose models in critical traits, such as sustaining therapeutic engagement, minimizing artificial or exaggerated linguistic patterns, and maintaining fidelity to PCT theoretical framework.