mlsanityData to 7 Oct 2026

SocialBench: Sociality Evaluation of Role-Playing Conversational Agents

arXiv 2403.13679 ↗First posted 20 Mar 2024cs.CLcited by 48

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 15 of its 31 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

15 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation2 Jan
  2. 2023Do LLM Agents Exhibit Social Behavior?23 Dec
  3. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct

Other references · 12

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  5. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  6. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  7. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  8. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 228
  9. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  10. 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 58
  11. 2022CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AIcited by 25
  12. 2023ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Humancited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 23; 2026: 17.

48 papers.

Sort:

By area:

  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  3. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  4. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  5. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 37
  6. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agentscited by 29
  7. 2025A Survey of Personalization: From RAG to Agentcited by 26
  8. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  9. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 21
  10. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  11. 2024Identity-Driven Hierarchical Role-Playing Agentscited by 18
  12. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
36 more
  1. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  2. 2025Adaptive Social Learning via Mode Policy Optimization for Language Agentscited by 15
  3. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  4. 2025Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoningcited by 14
  5. 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 10
  6. 2025The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agentscited by 7
  7. 2025Toward Embodied AGI: A Review of Embodied AI and the Road Aheadcited by 7
  8. 2025LIFELONG SOTOPIA: Evaluating Social Intelligence of Language Agents Over Lifelong Social Interactionscited by 7
  9. 2024A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Stylescited by 6
  10. 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
  11. 2024Thinking Before Speaking: A Role-playing Model with Mindsetcited by 5
  12. 2025If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMscited by 5
  13. 2026HumanLLM: Towards Personalized Understanding and Simulation of Human Naturecited by 5
  14. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  15. 2025Ella: Embodied Social Agents with Lifelong Memorycited by 4
  16. 2025SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agentscited by 4
  17. 2025ConsintBench: Evaluating Language Models on Real-World Consumer Intent Understandingcited by 2
  18. 2026SocialVeil: Probing Social Intelligence of Language Agents under Communication Barrierscited by 2
  19. 2024CharacterBench: Benchmarking Character Customization of Large Language Modelscited by 1
  20. 2025OPEN-THEATRE: An Open-Source Toolkit for LLM-based Interactive Dramacited by 1
  21. 2025AudioRole: An Audio Dataset for Character Role-Playing in Large Language Modelscited by 1
  22. 2026From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agentscited by 1
  23. 2026TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluationcited by 1
  24. 2026MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulationscited by 1
  25. 2025Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contextscited by 0
  26. 2026MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agentscited by 0
  27. 2026Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learningcited by 0
  28. 2026CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulatorscited by 0
  29. 2026BOOKMARKS: Efficient Active Storyline Memory for Role-playingcited by 0
  30. 2026ALSO: Adversarial Online Strategy Optimization for Social Agentscited by 0
  31. 2026EUDAIMONIA: Evaluating Undesirable Dynamics in AIcited by 0
  32. 2026Personality Anchoring for Social Simulation: Linking Personality, Social Behavior, and Interaction Success with LLM Agentscited by 0
  33. 2026Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulationcited by 0
  34. 2026Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluationcited by 0
  35. 2026GLARE: Generative Learning via Adversarial Reward Estimation For Social Dynamics Forecastingcited by 0
  36. 2026AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations
  2. 2024Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level
  3. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  4. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
  5. 2025How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks
  6. 2026Communicate-Predict-Act: Evaluating Social Intelligence of Agents
  7. 2026SCENE: Recognizing Social Norms and Sanctioning in Group Chats
  8. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios