mlsanityData to 14 Oct 2026

Role-Playing Evaluation for Large Language Models

arXiv 2505.13157 ↗First posted 19 May 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) demonstrate a notable capacity for adopting personas and engaging in role-playing. However, evaluating this ability presents significant challenges, as human assessments are resource-intensive and automated evaluations can be biased. To address this, we introduce Role-Playing Eval (RPEval), a novel benchmark designed to assess LLM role-playing capabilities across four key dimensions: emotional understanding, decision-making, moral alignment, and in-character consistency. This article details the construction of RPEval and presents baseline evaluations. Our code and dataset are available at https://github.com/yelboudouri/RPEval

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

18 of 24 references.

References · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  7. 2023Role-Play with Large Language Modelscited by 294
  8. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 175
  9. 2022Language Models as Agent Modelscited by 132
  10. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  11. 2021BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Datacited by 75
  12. 2019Personalized Dialogue Generation with Diversified Traitscited by 74
6 more
  1. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 66
  2. 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 60
  3. 2024Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversationscited by 36
  4. 2023Building Persona Consistent Dialogue Agents with Offline Reinforcement Learningcited by 14
  5. 2024Self-Prompt Tuning: Enable Autonomous Role-Playing in LLMscited by 12
  6. 2024Unveiling the Secrets of Engaging Conversations: Factors that Keep Users Hooked on Role-Playing Dialog Agentscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Memory-Driven Role-Playing: Evaluation and Enhancement of Persona Knowledge Utilization in LLMscited by 5
  2. 2025Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generationscited by 4
  3. 2025Moral Susceptibility and Robustness under Persona Role-Play in Large Language Modelscited by 4
  4. 2026PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systemscited by 4
  5. 2026EvolvingWorld: An Open-Schema Framework for Co-Evolving Role-Play Agents and World Model in Interactive Literary Worldcited by 1