# HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

arXiv: https://arxiv.org/abs/2601.19922 · HTML: https://mlsanity.com/paper/2601.19922/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 82 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2307.09042/) (2023)
- [Rehearsal: Simulating Conflict to Teach Conflict Resolution](https://mlsanity.com/paper/2309.12309/) (2023)
- [EmoBench: Evaluating the Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2402.12071/) (2024)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Large Language Models Produce Responses Perceived to be Empathic](https://mlsanity.com/paper/2403.18148/) (2024)
- [Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AI](https://mlsanity.com/paper/2309.12444/) (2023)
- [Chain of Empathy: Enhancing Empathetic Response of Large Language Models Based on Psychotherapy Models](https://mlsanity.com/paper/2311.04915/) (2023)
- [When Large Language Models are Reliable for Judging Empathic Communication](https://mlsanity.com/paper/2506.10150/) (2025)
- [Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models](https://mlsanity.com/paper/2409.20222/) (2024)
- [A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations](https://mlsanity.com/paper/2505.14106/) (2025)
- [Human-like Affective Cognition in Foundation Models](https://mlsanity.com/paper/2409.11733/) (2024)
- [Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models](https://mlsanity.com/paper/2310.14389/) (2023)
- [SENSE-7: Taxonomy and Dataset for Measuring User Perceptions of Empathy in Sustained Human-AI Conversations](https://mlsanity.com/paper/2509.16437/) (2025)
- [Steering Conversational Large Language Models for Long Emotional Support Conversations](https://mlsanity.com/paper/2402.10453/) (2024)
- [Is ChatGPT More Empathetic than Humans?](https://mlsanity.com/paper/2403.05572/) (2024)
- [Scoring with Large Language Models: A Study on Measuring Empathy of Responses in Dialogues](https://mlsanity.com/paper/2412.20264/) (2024)

## Built on by (most cited first, 13 of 13)

- [AI generates well-liked but templatic empathic responses](https://mlsanity.com/paper/2604.08479/) (2026)
- [Discourse Diversity in Multi-Turn Empathic Dialogue](https://mlsanity.com/paper/2604.11742/) (2026)
- [LifeSide: Benchmarking Agents as Lifelong Digital Companions](https://mlsanity.com/paper/2606.04660/) (2026)
- [EmoLLM: Appraisal-Grounded Cognitive-Emotional Co-Reasoning in Large Language Models](https://mlsanity.com/paper/2603.16553/) (2026)
- [Perfecting Human-AI Interaction at Clinical Scale. Turning Production Signals into Safer, More Human Conversations](https://mlsanity.com/paper/2603.29893/) (2026)
- [Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm](https://mlsanity.com/paper/2605.27914/) (2026)
- [NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs](https://mlsanity.com/paper/2605.29685/) (2026)
- [Can Induced Emotion Bias LLM Behaviors in Sequential Decision Making?](https://mlsanity.com/paper/2607.12631/) (2026)
- [Beyond Feeling Better: Capability-Sustaining Emotional Dialogue as a Longitudinal Research Paradigm](https://mlsanity.com/paper/2607.27851/) (2026)
- [Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue](https://mlsanity.com/paper/2608.26529/) (2026)
- [Empathy Is Steerable but Multi-Axial: Mechanism Geometry and Persona Effects in LLMs](https://mlsanity.com/paper/2609.15654/) (2026)
- [RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations](https://mlsanity.com/paper/2610.01780/) (2026)
- [RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models](https://mlsanity.com/paper/2610.09569/) (2026)

## Abstract (arXiv)

Supportive conversation depends on skills that go beyond language fluency, including reading emotions, adjusting tone, and navigating moments of resistance, frustration, or distress. Despite rapid progress in language models, we still lack a clear way to understand how their abilities in these interpersonal domains compare to those of humans. We introduce HEART, the first-ever framework that directly compares humans and LLMs on the same multi-turn emotional-support conversations. For each dialogue history, we pair human and model responses and evaluate them through blinded human raters and an ensemble of LLM-as-judge evaluators. All assessments follow a rubric grounded in interpersonal communication science across five dimensions: Human Alignment, Empathic Responsiveness, Attunement, Resonance, and Task-Following. HEART uncovers striking behavioral patterns. Several frontier models approach or surpass the average human responses in perceived empathy and consistency. At the same time, humans maintain advantages in adaptive reframing, tension-naming, and nuanced tone shifts, particularly in adversarial turns. Human and LLM-as-judge preferences align on about 80 percent of pairwise comparisons, matching inter-human agreement, and their written rationales emphasize similar HEART dimensions. This pattern suggests an emerging convergence in the criteria used to assess supportive quality. By placing humans and models on equal footing, HEART reframes supportive dialogue as a distinct capability axis, separable from general reasoning or linguistic fluency. It provides a unified empirical foundation for understanding where model-generated support aligns with human social judgment, where it diverges, and how affective conversational competence scales with model size.