# Self-Assessment Tests are Unreliable Measures of LLM Personality

arXiv: https://arxiv.org/abs/2309.08163 · HTML: https://mlsanity.com/paper/2309.08163/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 15 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [Estimating the Personality of White-Box Language Models](https://mlsanity.com/paper/2204.12000/) (2022)
- [Identifying and Manipulating the Personality Traits of Language Models](https://mlsanity.com/paper/2212.10276/) (2022)
- [Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs](https://mlsanity.com/paper/2305.14693/) (2023)
- [AI Text-to-Behavior: A Study In Steerability](https://mlsanity.com/paper/2308.07326/) (2023)

## Built on by (most cited first, 30 of 39)

- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics](https://mlsanity.com/paper/2406.14703/) (2024)
- [We're Different, We're the Same: Creative Homogeneity Across LLMs](https://mlsanity.com/paper/2501.19361/) (2025)
- [Personality Alignment of Large Language Models](https://mlsanity.com/paper/2408.11779/) (2024)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History](https://mlsanity.com/paper/2508.04826/) (2025)
- [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024)
- [Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications](https://mlsanity.com/paper/2505.00049/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations](https://mlsanity.com/paper/2601.09833/) (2026)
- [Evaluating Alignment of Behavioral Dispositions in LLMs](https://mlsanity.com/paper/2602.11328/) (2026)
- [Rediscovering the Latent Dimensions of Personality with Large Language Models as Trait Descriptors](https://mlsanity.com/paper/2409.09905/) (2024)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Towards Region-aware Bias Evaluation Metrics](https://mlsanity.com/paper/2406.16152/) (2024)
- [CAPE: Context-Aware Personality Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2508.20385/) (2025)
- [Beyond Self-Reports: Multi-Observer Agents for Personality Assessment in Large Language Models](https://mlsanity.com/paper/2504.08399/) (2025)
- [Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding](https://mlsanity.com/paper/2507.11198/) (2025)
- [Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need](https://mlsanity.com/paper/2510.11734/) (2025)
- [Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Reality](https://mlsanity.com/paper/2503.16457/) (2025)
- [Human Psychometric Questionnaires Mischaracterize LLM Behavior](https://mlsanity.com/paper/2509.10078/) (2025)
- [Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study](https://mlsanity.com/paper/2602.17262/) (2026)
- [Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact](https://mlsanity.com/paper/2606.20205/) (2026)
- [I'm Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk](https://mlsanity.com/paper/2504.00012/) (2025)
- [Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents](https://mlsanity.com/paper/2508.00742/) (2025)
- [Psychologically Enhanced AI Agents](https://mlsanity.com/paper/2509.04343/) (2025)
- [Machine individuality: Separating genuine idiosyncrasy from response bias in large language models](https://mlsanity.com/paper/2604.16755/) (2026)
- [Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior](https://mlsanity.com/paper/2606.12730/) (2026)
- [Efficient Safety Alignment of Language Models via Latent Personality Traits](https://mlsanity.com/paper/2607.07918/) (2026)
- [Evidence of conceptual mastery in the application of rules by Large Language Models](https://mlsanity.com/paper/2503.00992/) (2025)

## Abstract (arXiv)

As large language models (LLM) evolve in their capabilities, various recent studies have tried to quantify their behavior using psychological tools created to study human behavior. One such example is the measurement of "personality" of LLMs using self-assessment personality tests developed to measure human personality. Yet almost none of these works verify the applicability of these tests on LLMs. In this paper, we analyze the reliability of LLM personality scores obtained from self-assessment personality tests using two simple experiments. We first introduce the property of prompt sensitivity, where three semantically equivalent prompts representing three intuitive ways of administering self-assessment tests on LLMs are used to measure the personality of the same LLM. We find that all three prompts lead to very different personality scores, a difference that is statistically significant for all traits in a large majority of scenarios. We then introduce the property of option-order symmetry for personality measurement of LLMs. Since most of the self-assessment tests exist in the form of multiple choice question (MCQ) questions, we argue that the scores should also be robust to not just the prompt template but also the order in which the options are presented. This test unsurprisingly reveals that the self-assessment test scores are not robust to the order of the options. These simple tests, done on ChatGPT and three Llama2 models of different sizes, show that self-assessment personality tests created for humans are unreliable measures of personality in LLMs.