mlsanityData to 7 Oct 2026

Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference

arXiv 2608.26674 ↗First posted 27 Aug 2026cs.CL cs.AIcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 62 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

8 of 62 references.

References · 8

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  3. 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 58
  4. 2023ChatHaruhi: Reviving Anime Character in Reality via Large Language Modelcited by 57
  5. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 41
  6. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  7. 2026Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effectscited by 3
  8. 2026PsyCLIENT: Client Simulation via Conversational Trajectory Modeling for Trainee Practice and Model Evaluation in Mental Health Counselingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2024PersonaGym: Evaluating Persona Agents and LLMs
  2. 2025Improving Language Model Personas via Rationalization with Psychological Scaffolds
  3. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humans
  4. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
  5. 2025PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions
  6. 2026Do LLMs Experience an Internal Polylogue? Investigating Reasoning through the Lens of Personas
  7. 2025PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?
  8. 2025Can LLM Agents Maintain a Persona in Discourse?