mlsanityData to 14 Oct 2026

Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History

arXiv 2508.04826 ↗First posted 6 Aug 2025cs.CL cs.AIcited by 26

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models require consistent behavioral patterns for safe deployment, yet there are indications of large variability that may lead to an instable expression of personality traits in these models. We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25 open-source models (1B-685B parameters) across 2 million+ responses. Using traditional (BFI, SD3) and novel LLM-adapted personality questionnaires, we systematically vary model size, personas, reasoning modes, question order or paraphrasing, and conversation history. Our findings challenge fundamental assumptions: (1) Question reordering alone can introduce large shifts in personality measurements; (2) Scaling provides limited stability gains: even 400B+ models exhibit standard deviations >0.3 on 5-point scales; (3) Interventions expected to stabilize behavior, such as reasoning and inclusion of conversation history, can paradoxically increase variability; (4) Detailed persona instructions produce mixed effects, with misaligned personas showing significantly higher variability than the helpful assistant baseline; (5) The LLM-adapted questionnaires, despite their improved ecological validity, exhibit instability comparable to human-centric versions. This persistent instability across scales and mitigation strategies suggests that current LLMs lack the architectural foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that current alignment strategies may be inadequate.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 37 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

20 of 37 references.

References · 20

  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  3. 2024Alignment faking in large language modelscited by 360
  4. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  5. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  6. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  7. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  8. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  9. 2023Personality Traits in Large Language Modelscited by 169
  10. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  11. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  12. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
8 more
  1. 2025Persona Features Control Emergent Misalignmentcited by 86
  2. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 59
  3. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 49
  4. 2025Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancementcited by 45
  5. 2023Self-Assessment Tests are Unreliable Measures of LLM Personalitycited by 39
  6. 2023Challenging the Validity of Personality Tests for Large Language Modelscited by 25
  7. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 25
  8. 2024Lost in Translation: The Algorithmic Gap Between LMs and the Braincited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 21.

26 papers.

Sort:

By area:

  1. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  2. 2026The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Studycited by 5
  3. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  4. 2026Activation Steering for Aligned Open-ended Generation without Sacrificing Coherencecited by 4
  5. 2025Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Needcited by 3
  6. 2025Addressing Longstanding Challenges in Cognitive Science with Language Modelscited by 3
  7. 2025Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AIcited by 3
  8. 2026Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Modelscited by 3
  9. 2025Scaling behavior of large language models in emotional safety classification across sizes and taskscited by 1
  10. 2025Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contextscited by 1
  11. 2026Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detectioncited by 1
  12. 2026Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMscited by 1
14 more
  1. 2026Latent Personality Alignment: Improving Harmlessness Without Mentioning Harmscited by 1
  2. 2026Efficient Safety Alignment of Language Models via Latent Personality Traitscited by 1
  3. 2026Role Steering of Language Models for Social Simulationscited by 1
  4. 2026Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Controlcited by 1
  5. 2026The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AIcited by 0
  6. 2026InterviewSim: A Scalable Framework for Interview-Grounded Personality Simulationcited by 0
  7. 2026The Company You Keep: How LLMs Respond to Dark Triad Traitscited by 0
  8. 2026Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreakscited by 0
  9. 2026The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometricscited by 0
  10. 2026Personality Anchoring for Social Simulation: Linking Personality, Social Behavior, and Interaction Success with LLM Agentscited by 0
  11. 2026StabilityBench: Benchmarking Instability in LLMscited by 0
  12. 2026Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidencecited by 0
  13. 2026Emergi-PersonaOS: A Persona Agent Operating System for Situational Adaptation and Controllable Evolutioncited by 0
  14. 2026Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
  2. 2025Evaluating LLM Alignment on Personality Inference from Real-World Interview Data
  3. 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
  4. 2026PTCBENCH: Benchmarking Contextual Stability of Personality Traits in LLM Systems
  5. 2024Examining Identity Drift in Conversations of LLM Agents
  6. 2025Can LLM Agents Maintain a Persona in Discourse?
  7. 2026Personality Expression Across Contexts: Linguistic and Behavioral Variation in LLM Agents
  8. 2026When Personality Meets Quantization: A Layer-wise MBTI Analysis of Quantized LLMs