mlsanityData to 14 Oct 2026

EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World

arXiv 2601.01530 ↗First posted 4 Jan 2026cs.CL cs.HCcited by 3

Not on the map. Why?

Abstract (arXiv)

Current evaluation paradigms for emotional support conversations tend to reward generic empathetic responses, yet they fail to assess whether the support is genuinely personalized to users' unique psychological profiles and contextual needs. We introduce EmoHarbor, an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. EmoHarbor employs a Chain-of-Agent architecture that decomposes users' internal processes into three specialized roles, enabling agents to interact with supporters and complete assessments in a manner similar to human users. We instantiate this benchmark using 100 real-world user profiles that cover a diverse range of personality traits and situations, and define 10 evaluation dimensions of personalized support quality. Comprehensive evaluation of 20 advanced LLMs on EmoHarbor reveals a critical insight: while these models excel at generating empathetic responses, they consistently fail to tailor support to individual user contexts. This finding reframes the central challenge, shifting research focus from merely enhancing generic empathy to developing truly user-aware emotional support. EmoHarbor provides a reproducible and scalable framework to guide the development and evaluation of more nuanced and user-aware emotional support systems.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 4 Jan 2026.

Builds on

27 of 44 references.

References · 27

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  7. 2023Large Language Modelscited by 780
  8. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  9. 2021Towards Emotional Support Dialog Systemscited by 299
  10. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 178
  11. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 143
  12. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
15 more
  1. 2023SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversationscited by 110
  2. 2024CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counselingcited by 89
  3. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 88
  4. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  5. 2022Control Globally, Understand Locally: A Global-to-Local Hierarchical Graph Network for Emotional Support Conversationcited by 56
  6. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 48
  7. 2024ESCoT: Towards Interpretable Emotional Support Dialogue Systemscited by 46
  8. 2022PAL: Persona-Augmented Emotional Support Conversation Generationcited by 39
  9. 2025Customizing Emotional Support: How Do Individuals Construct and Interact With LLM-Powered Chatbotscited by 25
  10. 2024SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agentcited by 24
  11. 2023BatchEval: Towards Human-like Text Evaluationcited by 17
  12. 2025CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimizationcited by 10
  13. 2025From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignmentcited by 9
  14. 2025ESC-Judge: A Framework for Comparing Emotional Support Conversational Agentscited by 8
  15. 2025SocialSim: Towards Socialized Simulation of Emotional Support Conversationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026MemEmo: Evaluating Emotion in Memory Systems of Agentscited by 1
  2. 2026Listening to the Echo: User-Reaction Aware Policy Optimization via Scalar-Verbal Hybrid Reinforcement Learningcited by 1
  3. 2026A Responsive Present, a Shared Past, a Social Other: Teens' Overreliance on Companion AI Chatbotscited by 0