mlsanityData to 7 Oct 2026

PersonaGym: Evaluating Persona Agents and LLMs

arXiv 2407.18416 ↗First posted 25 Jul 2024cs.CL cs.AI cs.LGcited by 89

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Persona agents, which are LLM agents conditioned to act according to an assigned persona, enable contextually rich and user aligned interactions across domains like education and healthcare. However, evaluating how faithfully these agents adhere to their personas remains a significant challenge, particularly in free-form settings that demand consistency across diverse, persona-relevant environments. We introduce PersonaGym, the first dynamic evaluation framework for persona agents, and PersonaScore, a human-aligned automatic metric grounded in decision theory that enables comprehensive large-scale evaluation. Our evaluation of 10 leading LLMs across 200 personas and 10,000 questions reveals significant advancement opportunities. For example, GPT-4.1 had the exact same PersonaScore as LLaMA-3-8b despite being a more recent and advanced closed source model. Importantly, increased model size and complexity do not necessarily enhance persona agent capabilities, underscoring the need for algorithmic and architectural innovation toward faithful, performant persona agents.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 28 of its 49 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

28 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent21 Feb
  2. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation2 Jan
  3. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct
  4. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View3 Oct
  5. 2022Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters13 Nov

Other references · 23

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  6. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  7. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  8. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  9. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  10. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  11. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 177
  12. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
11 more
  1. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  2. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  3. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
  4. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  5. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  6. 2023AvalonBench: Evaluating LLMs Playing the Game of Avaloncited by 45
  7. 2024From Role-Play to Drama-Interaction: An LLM Solutioncited by 42
  8. 2022Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterancescited by 32
  9. 2024Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMscited by 11
  10. 2023Thespian: Multi-Character Text Role-Playing Game Agentscited by 6
  11. 2024Enhancing Supermarket Robot Interaction: A Multi-Level LLM Conversational Interface for Handling Diverse Customer Intentscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 41; 2026: 42.

89 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  2. 2024Beyond Browsing: API-Based Web Agentscited by 77
  3. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  4. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 44
  5. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  6. 2025PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactionscited by 30
  7. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  8. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 26
  9. 2024Examining Identity Drift in Conversations of LLM Agentscited by 25
  10. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  11. 2026Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversitycited by 25
  12. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
38 more
  1. 2025How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulationcited by 21
  2. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  3. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  4. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  5. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  6. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  7. 2024PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluationcited by 10
  8. 2025Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMscited by 9
  9. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  10. 2025Towards Multi-Agent Reasoning Systems for Collaborative Expertise Delegation: An Exploratory Design Studycited by 8
  11. 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
  12. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 8
  13. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulationcited by 8
  14. 2025TinyTroupe: An LLM-powered Multiagent Persona Simulation Toolkitcited by 7
  15. 2025HARBOR: Exploring Persona Dynamics in Multi-Agent Competitioncited by 6
  16. 2025MobileCity: An Efficient Framework for Large-Scale Urban Behavior Simulationcited by 5
  17. 2025From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilerscited by 5
  18. 2025MOA: Multi-Objective Alignment for Role-Playing Agentscited by 5
  19. 2024DiverseDialogue: A Methodology for Designing Chatbots with Human-Like Diversitycited by 4
  20. 2025AI Propaganda factories with language modelscited by 4
  21. 2025Moral Susceptibility and Robustness under Persona Role-Play in Large Language Modelscited by 4
  22. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  23. 2025Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generationcited by 3
  24. 2025Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Contextcited by 3
  25. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  26. 2025Evolution in Simulation: AI-Agent School with Dual Memory for High-Fidelity Educational Dynamicscited by 3
  27. 2025Do Persona-Infused LLMs Affect Performance in a Strategic Reasoning Game?cited by 3
  28. 2026The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomescited by 3
  29. 2026PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistencycited by 3
  30. 2026PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Modelscited by 3
  31. 2026RoleCDE:Benchmarking and Mitigating Role-Alignment Trade-offs in Role-Playing Agentscited by 3
  32. 2025The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theoriescited by 2
  33. 2025Beyond Single-Sentence Prompts: Upgrading Value Alignment Benchmarks with Dialogues and Storiescited by 2
  34. 2025Can Lessons From Human Teams Be Applied to Multi-Agent Systems? The Role of Structure, Diversity, and Interaction Dynamicscited by 2
  35. 2025PersonaAgent with GraphRAG: Community-Aware Knowledge Graphs for Personalized LLMcited by 2
  36. 2026Identifying and Mitigating Bottlenecks in Role-Playing Agents: A Systematic Study of Disentangling Character Profile Axescited by 2
  37. 2026Eval4Sim: An Evaluation Framework for Persona Simulationcited by 2
  38. 2025CSP: A Simulator For Multi-Agent Ranking Competitionscited by 1

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization
  2. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humans
  3. 2024Building Better AI Agents: A Provocation on the Utilisation of Persona in LLM-based Conversational Agents
  4. 2025PersonaAgent: Bridging Memory and Action for Personalized LLM Agents
  5. 2026Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM
  6. 2026Agentic Persona Generation with Critique-Refinement: An Industrial Evaluation
  7. 2026A Three-Tier Persona Vector for Controllable User Simulation in Agentic Evaluation
  8. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment