mlsanityData to 14 Oct 2026

CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation

arXiv 2401.01275 ↗First posted 2 Jan 2024cs.CLcited by 143

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Recently, the advent of large language models (LLMs) has revolutionized generative agents. Among them, Role-Playing Conversational Agents (RPCAs) attract considerable attention due to their ability to emotionally engage users. However, the absence of a comprehensive benchmark impedes progress in this field. To bridge this gap, we introduce CharacterEval, a Chinese benchmark for comprehensive RPCA assessment, complemented by a tailored high-quality dataset. The dataset comprises 1,785 multi-turn role-playing dialogues, encompassing 23,020 examples and featuring 77 characters derived from Chinese novels and scripts. It was carefully constructed, beginning with initial dialogue extraction via GPT-4, followed by rigorous human-led quality control, and enhanced with in-depth character profiles sourced from Baidu Baike. CharacterEval employs a multifaceted evaluation approach, encompassing thirteen targeted metrics on four dimensions. Comprehensive experiments on CharacterEval demonstrate that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. Source code, data source and reward model will be publicly accessible at https://github.com/morecry/CharacterEval.

Key facts. Cited by 143 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 21 of its 38 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

21 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  8. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 233
  9. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  10. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  11. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  12. 2019Personalized Dialogue Generation with Diversified Traitscited by 74
8 more
  1. 2019A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Datacited by 74
  2. 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 60
  3. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  4. 2020Low-Resource Knowledge-Grounded Dialogue Generationcited by 53
  5. 2022Less is More: Learning to Refine Dialogue History for Personalized Dialogue Generationcited by 47
  6. 2020Zero-Resource Knowledge-Grounded Dialogue Generationcited by 35
  7. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19
  8. 2019Reinforcement Learning for Personalized Dialogue Managementcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 36; 2025: 54; 2026: 53.

143 papers.

Sort:

By area:

  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  2. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  3. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 97
  4. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  5. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agentcited by 52
  6. 2024From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake Newscited by 51
  7. 2024SocialBench: Sociality Evaluation of Role-Playing Conversational Agentscited by 50
  8. 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 50
  9. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 48
  10. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  11. 2024From Role-Play to Drama-Interaction: An LLM Solutioncited by 42
  12. 2025Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profilescited by 41
38 more
  1. 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
  2. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
  3. 2025Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learningcited by 27
  4. 2025A Survey of Personalization: From RAG to Agentcited by 27
  5. 2024Emotional RAG: Enhancing Role-Playing Agents through Emotional Retrievalcited by 26
  6. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  7. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  8. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 22
  9. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  10. 2025How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulationcited by 22
  11. 2024MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agentscited by 21
  12. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  13. 2024Identity-Driven Hierarchical Role-Playing Agentscited by 18
  14. 2024SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction with 3D Autonomous Characterscited by 18
  15. 2026A Survey on LLM-based Conversational User Simulationcited by 18
  16. 2024Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playingcited by 17
  17. 2024ESC-Eval: Evaluating Emotion Support Conversations in Large Language Modelscited by 15
  18. 2025TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulationcited by 15
  19. 2025Reasoning Does Not Necessarily Improve Role-Playing Abilitycited by 14
  20. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  21. 2024BEYOND DIALOGUE: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Modelcited by 13
  22. 2025Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challengescited by 12
  23. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
  24. 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 11
  25. 2024Crafting Customisable Characters with LLMs: A Persona-Driven Role-Playing Agent Frameworkcited by 10
  26. 2024MINDECHO: Role-Playing Language Agents for Key Opinion Leaderscited by 10
  27. 2024PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluationcited by 10
  28. 2025Debt Collection Negotiations with Large Language Models: An Evaluation System and Optimizing Decision Making with Multi-Agentcited by 10
  29. 2025RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playingcited by 10
  30. 2025CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimizationcited by 10
  31. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 10
  32. 2026HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playingcited by 10
  33. 2025PsyPlay: Personality-Infused Role-Playing Conversational Agentscited by 9
  34. 2025Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMscited by 9
  35. 2025PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?cited by 9
  36. 2025MOOM: Maintenance, Organization and Optimization of Memory in Ultra-Long Role-Playing Dialoguescited by 8
  37. 2024Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agentscited by 7
  38. 2025RoleRAG: Enhancing LLM Role-Playing via Graph Guided Retrievalcited by 7

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models
  2. 2022Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters
  3. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment
  4. 2024Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?
  5. 2025RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following
  6. 2025Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Prompting
  7. 2026Identifying and Mitigating Bottlenecks in Role-Playing Agents: A Systematic Study of Disentangling Character Profile Axes
  8. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators