# SocialBench: Sociality Evaluation of Role-Playing Conversational Agents

arXiv: https://arxiv.org/abs/2403.13679 · HTML: https://mlsanity.com/paper/2403.13679/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [CharacterChat: Learning towards Conversational AI with Personalized Social Support](https://mlsanity.com/paper/2308.10278/) (2023)
- [Do LLM Agents Exhibit Social Behavior?](https://mlsanity.com/paper/2312.15198/) (2023)
- [CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI](https://mlsanity.com/paper/2205.14727/) (2022)
- [ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human](https://mlsanity.com/paper/2304.07849/) (2023)

## Built on by (most cited first, 30 of 50)

- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios](https://mlsanity.com/paper/2410.19346/) (2024)
- [SDPO: Segment-Level Direct Preference Optimization for Social Agents](https://mlsanity.com/paper/2501.01821/) (2025)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas](https://mlsanity.com/paper/2501.15427/) (2025)
- [TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models](https://mlsanity.com/paper/2405.18027/) (2024)
- [Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models](https://mlsanity.com/paper/2505.02847/) (2025)
- [Identity-Driven Hierarchical Role-Playing Agents](https://mlsanity.com/paper/2407.19412/) (2024)
- [SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents](https://mlsanity.com/paper/2502.15538/) (2025)
- [Cultural Learning-Based Culture Adaptation of Language Models](https://mlsanity.com/paper/2504.02953/) (2025)
- [Adaptive Social Learning via Mode Policy Optimization for Language Agents](https://mlsanity.com/paper/2505.02156/) (2025)
- [OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction](https://mlsanity.com/paper/2505.20277/) (2025)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [Too Good to be Bad: On the Failure of LLMs to Role-Play Villains](https://mlsanity.com/paper/2511.04962/) (2025)
- [Toward Embodied AGI: A Review of Embodied AI and the Road Ahead](https://mlsanity.com/paper/2505.14235/) (2025)
- [LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactions](https://mlsanity.com/paper/2506.12666/) (2025)
- [HumanLLM: Towards Personalized Understanding and Simulation of Human Nature](https://mlsanity.com/paper/2601.15793/) (2026)
- [The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents](https://mlsanity.com/paper/2502.20757/) (2025)
- [A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Styles](https://mlsanity.com/paper/2411.02457/) (2024)
- [Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games](https://mlsanity.com/paper/2604.11741/) (2026)
- [Thinking Before Speaking: A Role-playing Model with Mindset](https://mlsanity.com/paper/2409.13752/) (2024)
- [If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs](https://mlsanity.com/paper/2503.23514/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [Ella: Embodied Social Agents with Lifelong Memory](https://mlsanity.com/paper/2506.24019/) (2025)
- [SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents](https://mlsanity.com/paper/2508.02013/) (2025)
- [ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understanding](https://mlsanity.com/paper/2510.13499/) (2025)
- [SocialVeil: Probing Social Intelligence of Language Agents under Communication Barriers](https://mlsanity.com/paper/2602.05115/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench.