mlsanityData to 7 Oct 2026

Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents

arXiv 2502.13012 ↗First posted 18 Feb 2025cs.HC cs.CLcited by 22

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Role-Playing Agent (RPA) is an increasingly popular type of LLM Agent that simulates human-like behaviors in a variety of tasks. However, evaluating RPAs is challenging due to diverse task requirements and agent designs. This paper proposes an evidence-based, actionable, and generalizable evaluation design guideline for LLM-based RPA by systematically reviewing 1,676 papers published between Jan. 2021 and Dec. 2024. Our analysis identifies six agent attributes, seven task attributes, and seven evaluation metrics from existing literature. Based on these findings, we present an RPA evaluation design guideline to help researchers develop more systematic and consistent evaluation methods.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 63 of its 146 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

63 of 146 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents4 Dec
  2. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals15 Nov
  3. 2024The Stepwise Deception: Simulating the Evolution from True News to Fake News with LLM Agents24 Oct
  4. 2024SRAP-Agent: Simulating and Optimizing Scarce Resource Allocation Policy with LLM-based Agent18 Oct
  5. 2024I Want to Break Free! Persuasion and Anti-Social Behavior of LLMs in Multi-Agent Settings with Social Hierarchy9 Oct
  6. 2024ERABAL: Enhancing Role-Playing Agents through Boundary-Aware Learning23 Sep
  7. 2024AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents15 Aug
  8. 2024Self-Emotion Blended Dialogue Generation in Social Simulation Agents3 Aug
  9. 2024Identity-Driven Hierarchical Role-Playing Agents28 Jul
  10. 2024Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View23 May
  11. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents25 Apr
  12. 2024Emergence of Social Norms in Generative Agent Societies: Principles and Architecture13 Mar
8 more
  1. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents19 Feb
  2. 2024Simulating Human Strategic Behavior: Comparing Single and Multi-agent LLMs13 Feb
  3. 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models5 Feb
  4. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation2 Jan
  5. 2023Do LLM Agents Exhibit Social Behavior?23 Dec
  6. 2023Simulating Opinion Dynamics with Networks of LLM-based Agents16 Nov
  7. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct
  8. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming10 Oct

Other references · 43

  1. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  2. 2024Understanding the planning of LLM agents: A surveycited by 414
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  5. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  6. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  7. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  8. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  9. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  10. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  11. 2023Lyfe Agents: Generative agents for low-cost real-time social interactionscited by 49
  12. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
31 more
  1. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
  2. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  3. 2024Y Social: an LLM-powered Social Media Digital Twincited by 43
  4. 2024Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactionscited by 43
  5. 2024From Role-Play to Drama-Interaction: An LLM Solutioncited by 42
  6. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 37
  7. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  8. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  9. 2024AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Gamecited by 26
  10. 2024MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agentscited by 21
  11. 2024Proxona: Supporting Creators' Sensemaking and Ideation with LLM-Powered Audience Personascited by 19
  12. 2024Human Simulacra: Benchmarking the Personification of Large Language Modelscited by 17
  13. 2024Plurals: A System for Guiding LLMs Via Simulated Social Ensemblescited by 17
  14. 2024ESC-Eval: Evaluating Emotion Support Conversations in Large Language Modelscited by 15
  15. 2023Improving Interpersonal Communication by Simulating Audiences with Language Modelscited by 14
  16. 2024Depression Diagnosis Dialogue Simulation: Self-improving Psychiatrist with Tertiary Memorycited by 13
  17. 2024Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMscited by 11
  18. 2024Are LLMs Socially Adaptive? Contrasting Belief Evolution in Large Language Models and Humanscited by 10
  19. 2024How Personality Traits Influence Negotiation Outcomes? A Simulation based on Large Language Modelscited by 9
  20. 2023Heterogeneous Value Alignment Evaluation for Large Language Modelscited by 7
  21. 2024Why am I seeing this: Democratizing End User Auditing for Online Content Recommendationscited by 7
  22. 2024CogGPT: Unleashing the Power of Cognitive Dynamics on Large Language Modelscited by 6
  23. 2024AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behaviorcited by 6
  24. 2024PersLLM: A Personified Training Approach for Large Language Modelscited by 6
  25. 2024Using Large Language Models to Create AI Personas for Replication, Generalization and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findingscited by 4
  26. 2024DiverseDialogue: A Methodology for Designing Chatbots with Human-Like Diversitycited by 4
  27. 2025Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessmentcited by 4
  28. 2024Simulating Family Conversations using LLMs: Demonstration of Parenting Stylescited by 3
  29. 2024SAUCE: Synchronous and Asynchronous User-Customizable Environment for Multi-Agent LLM Interactioncited by 3
  30. 2024Scattering-Passive Structure-Preserving Finite Element Method for the Boundary Controlled Transport Equation with a Moving Meshcited by 1
  31. 2024Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 8.

22 papers.

Sort:

By area:

  1. 2025Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Datacited by 27
  2. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 26
  3. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 26
  4. 2025AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agentscited by 19
  5. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  6. 2026Does Socialization Emerge in AI Agent Society? A Case Study of Moltbookcited by 13
  7. 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
  8. 2025Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memorycited by 6
  9. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
  10. 2025Dynamic Context Adaptation for Consistent Role-Playing Agents with Retrieval-Augmented Generationscited by 4
  11. 2025Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaborationcited by 4
  12. 2025Detecting Perspective Shifts in Multi-agent Systemscited by 4
10 more
  1. 2025DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfacescited by 3
  2. 2026The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomescited by 3
  3. 2026Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effectscited by 3
  4. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agentscited by 2
  5. 2026Investigating Learner-Aware Design of LLM-Generated Educational Feedbackcited by 1
  6. 2026Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agentscited by 1
  7. 2025Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contextscited by 0
  8. 2025Beyond Permissions: Investigating Mobile Personalization with Simulated Personascited by 0
  9. 2026Backdoor Learning in Language Models and Vision-Language Modelscited by 0
  10. 2026One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents
  2. 2026Role-Playing Agents Driven by Large Language Models: Current Status, Challenges, and Future Trends
  3. 2026Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
  4. 2023LARP: Language-Agent Role Play for Open-World Games
  5. 2026RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agents
  6. 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
  7. 2024A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Styles
  8. 2025FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline