mlsanityData to 14 Oct 2026

InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews

arXiv 2310.17976 ↗First posted 27 Oct 2023cs.CLcited by 179

Not on the map. Why?

Cited by 51 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%.

Key facts. Cited by 179 papers in the map (data through 14 Oct 2026). Builds on 15 of its 47 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

15 of 47 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023Mistral 7Bcited by 3,164
  5. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  6. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  7. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  8. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  9. 2023The Self-Perception and Political Biases of ChatGPTcited by 66
  10. 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 60
  11. 2023Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Modelscited by 56
  12. 2022Estimating the Personality of White-Box Language Modelscited by 48
3 more
  1. 2023BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialoguescited by 34
  2. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  3. 2023LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streamingcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 53; 2025: 70; 2026: 56.

179 papers.

Sort:

By area:

  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  2. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  3. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 143
  4. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 113
  5. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  6. 2024On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agentscited by 99
  7. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 97
  8. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  9. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 66
  10. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 59
  11. 2024From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake Newscited by 51
  12. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 48
38 more
  1. 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
  2. 2024CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worldscited by 38
  3. 2024On the Structural Memory of LLM Agentscited by 37
  4. 2024Personality Alignment of Large Language Modelscited by 36
  5. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  6. 2024Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmissioncited by 35
  7. 2024Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?cited by 34
  8. 2025A Survey of Personalization: From RAG to Agentcited by 27
  9. 2024Emotional RAG: Enhancing Role-Playing Agents through Emotional Retrievalcited by 26
  10. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  11. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  12. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  13. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 22
  14. 2025How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulationcited by 22
  15. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  16. 2024MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agentscited by 21
  17. 2024RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systemscited by 21
  18. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  19. 2025AnnaAgent: Dynamic Evolution Agent System with Multi-Session Memory for Realistic Seeker Simulationcited by 21
  20. 2024Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Modelscited by 20
  21. 2024How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHOcited by 19
  22. 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
  23. 2024Mitigating Hallucination in Fictional Character Role-Playcited by 18
  24. 2024Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Datacited by 17
  25. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  26. 2024ESC-Eval: Evaluating Emotion Support Conversations in Large Language Modelscited by 15
  27. 2025MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQLcited by 15
  28. 2024Do LLMs have Consistent Values?cited by 14
  29. 2025Exploring the Impact of Personality Traits on LLM Bias and Toxicitycited by 14
  30. 2025Reasoning Does Not Necessarily Improve Role-Playing Abilitycited by 14
  31. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  32. 2025Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoningcited by 14
  33. 2025The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societiescited by 14
  34. 2024BEYOND DIALOGUE: A Profile-Dialogue Alignment Framework Towards General Role-Playing Language Modelcited by 13
  35. 2024Measuring Human and AI Values Based on Generative Psychometrics with Large Language Modelscited by 13
  36. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  37. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
  38. 2024Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Modelscited by 11

50 most cited of 179 All 179 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents
  2. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humans
  3. 2025PsyPlay: Personality-Infused Role-Playing Conversational Agents
  4. 2026Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
  5. 2024On the Decision-Making Abilities in Role-Playing using Large Language Models
  6. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
  7. 2026RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents
  8. 2026Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference