# EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues

arXiv: https://arxiv.org/abs/2301.12510 · HTML: https://mlsanity.com/paper/2301.12510/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 29 Jan 2023.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support](https://mlsanity.com/paper/2009.08441/) (2020)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Modeling Empathy and Distress in Reaction to News Stories](https://mlsanity.com/paper/1808.10399/) (2018)
- [Learning Word Ratings for Empathy and Distress from Document-Level User Responses](https://mlsanity.com/paper/1912.01079/) (2019)
- [Attention-based Modeling for Emotion Detection and Classification in Textual Conversations](https://mlsanity.com/paper/1906.07020/) (2019)

## Abstract (arXiv)

Measuring empathy in conversation can be challenging, as empathy is a complex and multifaceted psychological construct that involves both cognitive and emotional components. Human evaluations can be subjective, leading to inconsistent results. Therefore, there is a need for an automatic method for measuring empathy that reduces the need for human evaluations. In this paper, we proposed a novel approach EMP-EVAL, a simple yet effective automatic empathy evaluation method. The proposed technique takes the influence of Emotion, Cognitive and Emotional empathy. To the best knowledge, our work is the first to systematically measure empathy without the human-annotated provided scores. Experimental results demonstrate that our metrics can correlate with human preference, achieving comparable results with human judgments.