# Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

arXiv: https://arxiv.org/abs/2603.03915 · HTML: https://mlsanity.com/paper/2603.03915/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Character-LLM: A Trainable Agent for Role-Playing](https://mlsanity.com/paper/2310.10158/) (2023)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation](https://mlsanity.com/paper/2401.01275/) (2024)
- [Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://mlsanity.com/paper/2401.12954/) (2024)
- [Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://mlsanity.com/paper/2401.12474/) (2024)
- [CharacterChat: Learning towards Conversational AI with Personalized Social Support](https://mlsanity.com/paper/2308.10278/) (2023)
- [CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds](https://mlsanity.com/paper/2412.05631/) (2024)
- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents](https://mlsanity.com/paper/2502.13012/) (2025)
- [MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents](https://mlsanity.com/paper/2408.04203/) (2024)
- [OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction](https://mlsanity.com/paper/2505.20277/) (2025)
- [BEYOND DIALOGUE: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Model](https://mlsanity.com/paper/2408.10903/) (2024)
- [IMPersona: Evaluating Individual Level LM Impersonation](https://mlsanity.com/paper/2504.04332/) (2025)
- [MINDECHO: Role-Playing Language Agents for Key Opinion Leaders](https://mlsanity.com/paper/2407.05305/) (2024)
- [CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization](https://mlsanity.com/paper/2508.09074/) (2025)
- [Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing](https://mlsanity.com/paper/2409.11726/) (2024)
- [HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing](https://mlsanity.com/paper/2511.08017/) (2025)
- [CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards](https://mlsanity.com/paper/2507.17147/) (2025)
- [Orca: Enhancing Role-Playing Abilities of Large Language Models by Integrating Personality Traits](https://mlsanity.com/paper/2411.10006/) (2024)

## Built on by (most cited first, 3 of 3)

- [Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference](https://mlsanity.com/paper/2608.26674/) (2026)
- [Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents](https://mlsanity.com/paper/2609.13637/) (2026)
- [From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers](https://mlsanity.com/paper/2609.21349/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs). However, current evaluation frameworks rely heavily on well-known fictional characters, raising a critical concern: models may be leveraging their internal training memory of these characters rather than demonstrating role-playing capabilities. This reliance often leads to significant performance degradation when RPAs encounter unseen or out-of-distribution personas. To address this, we propose a more rigorous evaluation protocol designed to decouple role-playing proficiency from character recognition. Our experiments across multiple benchmarks demonstrate that anonymizing characters degrades performance, confirming that name exposure provides implicit cues that mask a model's true capability. To mitigate this, we investigate diverse personality augmentation as a method to enhance role fidelity in anonymous settings. We systematically analyze the impact of various personality-description methods on agent behavior and consistency. Our results show that incorporating personality information consistently improves RPA performance. This work establishes a more equitable evaluation standard and validates a scalable, personality-enhanced framework for constructing robust RPAs.