# Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs

arXiv: https://arxiv.org/abs/2305.14693 · HTML: https://mlsanity.com/paper/2305.14693/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Identifying and Manipulating the Personality Traits of Language Models](https://mlsanity.com/paper/2212.10276/) (2022)

## Built on by (most cited first, 30 of 32)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics](https://mlsanity.com/paper/2406.14703/) (2024)
- [Self-Assessment Tests are Unreliable Measures of LLM Personality](https://mlsanity.com/paper/2309.08163/) (2023)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [The Better Angels of Machine Personality: How Personality Relates to LLM Safety](https://mlsanity.com/paper/2407.12344/) (2024)
- [ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models](https://mlsanity.com/paper/2406.04214/) (2024)
- [Revisiting the Reliability of Psychological Scales on Large Language Models](https://mlsanity.com/paper/2305.19926/) (2023)
- [Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models](https://mlsanity.com/paper/2409.12106/) (2024)
- [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024)
- [Identifying Multiple Personalities in Large Language Models with External Evaluation](https://mlsanity.com/paper/2402.14805/) (2024)
- [Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications](https://mlsanity.com/paper/2505.00049/) (2025)
- [Know Thyself? On the Incapability and Implications of AI Self-Recognition](https://mlsanity.com/paper/2510.03399/) (2025)
- [Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales](https://mlsanity.com/paper/2409.19655/) (2024)
- [Evaluating the Efficacy of LLMs to Emulate Realistic Human Personalities](https://mlsanity.com/paper/2402.14879/) (2024)
- [Dynamic Generation of Personalities with Large Language Models](https://mlsanity.com/paper/2404.07084/) (2024)
- [AI Will Always Love You: Studying Implicit Biases in Romantic AI Companions](https://mlsanity.com/paper/2502.20231/) (2025)
- [CAPE: Context-Aware Personality Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2508.20385/) (2025)
- [Personality-Guided Code Generation Using Large Language Models](https://mlsanity.com/paper/2411.00006/) (2024)
- [Profile-LLM: Dynamic Profile Optimization for Realistic Personality Expression in LLMs](https://mlsanity.com/paper/2511.19852/) (2025)
- [Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Reality](https://mlsanity.com/paper/2503.16457/) (2025)
- [Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengths](https://mlsanity.com/paper/2506.02481/) (2025)
- [Context-Value-Action Architecture for Value-Driven Large Language Model Agents](https://mlsanity.com/paper/2604.05939/) (2026)
- [Humanity in AI: Detecting the Personality of Large Language Models](https://mlsanity.com/paper/2410.08545/) (2024)
- [Persona Dynamics: Unveiling the Impact of Personality Traits on Agents in Text-Based Games](https://mlsanity.com/paper/2504.06868/) (2025)
- [Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy](https://mlsanity.com/paper/2511.19872/) (2025)
- [Augmenting Dialog with Think-Aloud Utterances for Modeling Individual Personality Traits by LLM](https://mlsanity.com/paper/2510.09158/) (2025)
- [MindShift: Analyzing Language Models' Reactions to Psychological Prompts](https://mlsanity.com/paper/2512.09149/) (2025)
- [Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark](https://mlsanity.com/paper/2603.17838/) (2026)

## Abstract (arXiv)

Have Large Language Models (LLMs) developed a personality? The short answer is a resounding "We Don't Know!". In this paper, we show that we do not yet have the right tools to measure personality in language models. Personality is an important characteristic that influences behavior. As LLMs emulate human-like intelligence and performance in various tasks, a natural question to ask is whether these models have developed a personality. Previous works have evaluated machine personality through self-assessment personality tests, which are a set of multiple-choice questions created to evaluate personality in humans. A fundamental assumption here is that human personality tests can accurately measure personality in machines. In this paper, we investigate the emergence of personality in five LLMs of different sizes ranging from 1.5B to 30B. We propose the Option-Order Symmetry property as a necessary condition for the reliability of these self-assessment tests. Under this condition, the answer to self-assessment questions is invariant to the order in which the options are presented. We find that many LLMs personality test responses do not preserve option-order symmetry. We take a deeper look at LLMs test responses where option-order symmetry is preserved to find that in these cases, LLMs do not take into account the situational statement being tested and produce the exact same answer irrespective of the situation being tested. We also identify the existence of inherent biases in these LLMs which is the root cause of the aforementioned phenomenon and makes self-assessment tests unreliable. These observations indicate that self-assessment tests are not the correct tools to measure personality in LLMs. Through this paper, we hope to draw attention to the shortcomings of current literature in measuring personality in LLMs and call for developing tools for machine personality measurement.