# Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History

arXiv: https://arxiv.org/abs/2508.04826 · HTML: https://mlsanity.com/paper/2508.04826/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation](https://mlsanity.com/paper/2311.03348/) (2023)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025)
- [Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics](https://mlsanity.com/paper/2406.14703/) (2024)
- [Large Language Models as Superpositions of Cultural Perspectives](https://mlsanity.com/paper/2307.07870/) (2023)
- [Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement](https://mlsanity.com/paper/2505.08245/) (2025)
- [Self-Assessment Tests are Unreliable Measures of LLM Personality](https://mlsanity.com/paper/2309.08163/) (2023)
- [Challenging the Validity of Personality Tests for Large Language Models](https://mlsanity.com/paper/2311.05297/) (2023)
- [Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions](https://mlsanity.com/paper/2505.17120/) (2025)
- [Lost in Translation: The Algorithmic Gap Between LMs and the Brain](https://mlsanity.com/paper/2407.04680/) (2024)

## Built on by (most cited first, 26 of 26)

- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study](https://mlsanity.com/paper/2605.20767/) (2026)
- [ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions](https://mlsanity.com/paper/2605.24279/) (2026)
- [Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence](https://mlsanity.com/paper/2604.08169/) (2026)
- [Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need](https://mlsanity.com/paper/2510.11734/) (2025)
- [Addressing Longstanding Challenges in Cognitive Science with Language Models](https://mlsanity.com/paper/2511.00206/) (2025)
- [Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI](https://mlsanity.com/paper/2511.00230/) (2025)
- [Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models](https://mlsanity.com/paper/2604.10733/) (2026)
- [Scaling behavior of large language models in emotional safety classification across sizes and tasks](https://mlsanity.com/paper/2509.04512/) (2025)
- [Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts](https://mlsanity.com/paper/2512.03704/) (2025)
- [Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detection](https://mlsanity.com/paper/2601.19245/) (2026)
- [Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs](https://mlsanity.com/paper/2604.11120/) (2026)
- [Latent Personality Alignment: Improving Harmlessness Without Mentioning Harms](https://mlsanity.com/paper/2605.08496/) (2026)
- [Efficient Safety Alignment of Language Models via Latent Personality Traits](https://mlsanity.com/paper/2607.07918/) (2026)
- [Role Steering of Language Models for Social Simulations](https://mlsanity.com/paper/2608.00023/) (2026)
- [Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control](https://mlsanity.com/paper/2608.10703/) (2026)
- [The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI](https://mlsanity.com/paper/2602.17127/) (2026)
- [InterviewSim: A Scalable Framework for Interview-Grounded Personality Simulation](https://mlsanity.com/paper/2602.20294/) (2026)
- [The Company You Keep: How LLMs Respond to Dark Triad Traits](https://mlsanity.com/paper/2603.04299/) (2026)
- [Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreaks](https://mlsanity.com/paper/2604.18510/) (2026)
- [The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics](https://mlsanity.com/paper/2606.03357/) (2026)
- [Personality Anchoring for Social Simulation: Linking Personality, Social Behavior, and Interaction Success with LLM Agents](https://mlsanity.com/paper/2606.06936/) (2026)
- [StabilityBench: Benchmarking Instability in LLMs](https://mlsanity.com/paper/2607.20558/) (2026)
- [Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence](https://mlsanity.com/paper/2608.14903/) (2026)
- [Emergi-PersonaOS: A Persona Agent Operating System for Situational Adaptation and Controllable Evolution](https://mlsanity.com/paper/2609.27417/) (2026)
- [Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations](https://mlsanity.com/paper/2610.19817/) (2026)

## Abstract (arXiv)

Large language models require consistent behavioral patterns for safe deployment, yet there are indications of large variability that may lead to an instable expression of personality traits in these models. We present PERSIST (PERsonality Stability in Synthetic Text), a comprehensive evaluation framework testing 25 open-source models (1B-685B parameters) across 2 million+ responses. Using traditional (BFI, SD3) and novel LLM-adapted personality questionnaires, we systematically vary model size, personas, reasoning modes, question order or paraphrasing, and conversation history. Our findings challenge fundamental assumptions: (1) Question reordering alone can introduce large shifts in personality measurements; (2) Scaling provides limited stability gains: even 400B+ models exhibit standard deviations >0.3 on 5-point scales; (3) Interventions expected to stabilize behavior, such as reasoning and inclusion of conversation history, can paradoxically increase variability; (4) Detailed persona instructions produce mixed effects, with misaligned personas showing significantly higher variability than the helpful assistant baseline; (5) The LLM-adapted questionnaires, despite their improved ecological validity, exhibit instability comparable to human-centric versions. This persistent instability across scales and mitigation strategies suggests that current LLMs lack the architectural foundations for genuine behavioral consistency. For safety-critical applications requiring predictable behavior, these findings indicate that current alignment strategies may be inadequate.