mlsanityData to 14 Oct 2026

H2HTalk: Evaluating Large Language Models as Emotional Companion

arXiv 2507.03543 ↗First posted 4 Jul 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

As digital emotional support needs grow, Large Language Model companions offer promising authentic, always-available empathy, though rigorous evaluation lags behind model advancement. We present Heart-to-Heart Talk (H2HTalk), a benchmark assessing companions across personality development and empathetic interaction, balancing emotional intelligence with linguistic fluency. H2HTalk features 4,650 curated scenarios spanning dialogue, recollection, and itinerary planning that mirror real-world support conversations, substantially exceeding previous datasets in scale and diversity. We incorporate a Secure Attachment Persona (SAP) module implementing attachment-theory principles for safer interactions. Benchmarking 50 LLMs with our unified protocol reveals that long-horizon planning and memory retention remain key challenges, with models struggling when user needs are implicit or evolve mid-conversation. H2HTalk establishes the first comprehensive benchmark for emotionally intelligent companions. We release all materials to advance development of LLMs capable of providing meaningful and safe psychological support.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

25 of 51 references.

References · 25

  1. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  5. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  6. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  7. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  8. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  9. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  10. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 175
  11. 2024Large Multimodal Agents: A Surveycited by 105
  12. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
13 more
  1. 2024CogBench: a large language model walks into a psychology labcited by 63
  2. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 59
  3. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  4. 2024Current state of LLM Risks and AI Guardrailscited by 46
  5. 2024Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactionscited by 43
  6. 2023CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Modelscited by 34
  7. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  8. 2024C-ICL: Contrastive In-context Learning for Information Extractioncited by 23
  9. 2024Evaluating Large Language Models with Psychometricscited by 22
  10. 2024Large Language Model for Mental Health: A Systematic Reviewcited by 14
  11. 2023NarrativePlay: Interactive Narrative Understandingcited by 10
  12. 2024New Intent Discovery with Attracting and Dispersing Prototypecited by 6
  13. 2024Towards Real-world Scenario: Imbalanced New Intent Discoverycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 11
  2. 2026Perfecting Human-AI Interaction at Clinical Scale. Turning Production Signals into Safer, More Human Conversationscited by 0
  3. 2026Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theorycited by 0