# CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship

arXiv: https://arxiv.org/abs/2608.02046 · HTML: https://mlsanity.com/paper/2608.02046/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations](https://mlsanity.com/paper/2311.00273/) (2023)
- [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021)
- [CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling](https://mlsanity.com/paper/2405.16433/) (2024)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning](https://mlsanity.com/paper/2512.01282/) (2025)
- [When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation](https://mlsanity.com/paper/2510.19032/) (2025)
- [INTIMA: A Benchmark for Human-AI Companionship Behavior](https://mlsanity.com/paper/2508.09998/) (2025)
- [Detecting Emotional Dynamic Trajectories: An Evaluation Framework for Emotional Support in Language Models](https://mlsanity.com/paper/2511.09003/) (2025)
- [EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management](https://mlsanity.com/paper/2606.15532/) (2026)
- [Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers](https://mlsanity.com/paper/2601.07698/) (2026)
- [Dual Optimal: Make Your LLM Peer-like with Dignity](https://mlsanity.com/paper/2604.00979/) (2026)
- [Dynamic In-Group Persona Generation for Enhancing Human-AI Rapport](https://mlsanity.com/paper/2606.18256/) (2026)

## Built on by (most cited first, 5 of 5)

- [Disclosure-Gated User Simulation for Companion-Agent Evaluation](https://mlsanity.com/paper/2609.00982/) (2026)
- [Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent](https://mlsanity.com/paper/2609.38604/) (2026)
- [RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations](https://mlsanity.com/paper/2610.01780/) (2026)
- [RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models](https://mlsanity.com/paper/2610.09569/) (2026)
- [CompanionSafe: From Multi-turn Safety Evaluation to Guardrails for AI Companions](https://mlsanity.com/paper/2610.18095/) (2026)

## Abstract (arXiv)

LLM companions are deployed at scale in personally consequential settings, yet poorly evaluated. Existing benchmarks use hand-authored scenarios and prompted simulators, aggregate empathy into one score, and overlook judge biases such as same-family favoritism and scale drift. We introduce CompanionBench, an interactive bilingual benchmark. To our knowledge, it is the first companion benchmark to ground both its scenarios and a trained user simulator in de-identified real-world data. A hidden disclosure gate branches each persona's trajectory on the agent's own behavior, controlling the interaction state space without scripting dialogue. We operationalize ten capabilities derived from 25 theories across psychology and counseling, four of them not graded explicitly by prior work: holding ambiguity, selfobject responsiveness, positive resonance and calibrated challenge. Agents are assessed on two complementary axes: a subjective ten-capability rubric and a deterministic measure of whether deeper disclosure was earned. A cross-family panel dilutes same-family favoritism; an Item Response Theory model separates agent quality from judge severity. Theory fixes what to measure and how personas are structured; real data supply events, history, and profiles -- coverage from theory, authenticity from data. Rankings are reproducible in both languages (rho = 0.996 ZH / 0.953 EN). Evaluating 28 agents reveals capability-level differences obscured by aggregate scores. Emotion regulation and calibrated challenge remain common weaknesses; holding ambiguity discriminates most. Role-play agents rank near the bottom: immersion does not imply relational competence. Across agents, the dominant failure mode is substituting surface warmth for substantive relational support. We will release 500 Chinese-English parallel pairs and the evaluation code.