# Identifying Multiple Personalities in Large Language Models with External Evaluation

arXiv: https://arxiv.org/abs/2402.14805 · HTML: https://mlsanity.com/paper/2402.14805/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [Estimating the Personality of White-Box Language Models](https://mlsanity.com/paper/2204.12000/) (2022)
- [Identifying and Manipulating the Personality Traits of Language Models](https://mlsanity.com/paper/2212.10276/) (2022)
- [Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs](https://mlsanity.com/paper/2305.14693/) (2023)
- [Orders Are Unwanted: Dynamic Deep Graph Convolutional Network for Personality Detection](https://mlsanity.com/paper/2212.01515/) (2022)
- [AI Text-to-Behavior: A Study In Steerability](https://mlsanity.com/paper/2308.07326/) (2023)
- [An Attention-Based Denoising Framework for Personality Detection in Social Media Texts](https://mlsanity.com/paper/2311.09945/) (2023)

## Built on by (most cited first, 10 of 10)

- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [The Better Angels of Machine Personality: How Personality Relates to LLM Safety](https://mlsanity.com/paper/2407.12344/) (2024)
- [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024)
- [Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications](https://mlsanity.com/paper/2505.00049/) (2025)
- [PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements](https://mlsanity.com/paper/2408.03337/) (2024)
- [Behavioral Fingerprinting of Large Language Models](https://mlsanity.com/paper/2509.04504/) (2025)
- [Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Reality](https://mlsanity.com/paper/2503.16457/) (2025)
- [Psychological Profiling in Cybersecurity: A Look at LLMs and Psycholinguistic Features](https://mlsanity.com/paper/2406.18783/) (2024)
- [PHORECAST: Enabling AI Understanding of Public Health Outreach Across Populations](https://mlsanity.com/paper/2510.02535/) (2025)
- [Multiple latent orderings better predict language model preferences](https://mlsanity.com/paper/2609.22170/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) are integrated with human daily applications rapidly, many societal and ethical concerns are raised regarding the behavior of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their personalities. Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans. Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs. In this paper, we investigate LLM personalities using an alternate personality measurement method, which we refer to as the external evaluation method, where instead of prompting LLMs with multiple-choice questions in the Likert scale, we evaluate LLMs' personalities by analyzing their responses toward open-ended situational questions using an external machine learning model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor that outperforms the state-of-the-art models as the tool to analyze LLMs' responses. Then, we prompt the LLMs with situational questions and ask them to generate Twitter posts and comments, respectively, in order to assess their personalities when playing two different roles. Using the external personality evaluation method, we identify that the obtained personality types for LLMs are significantly different when generating posts versus comments, whereas humans show a consistent personality profile in these two different situations. This shows that LLMs can exhibit different personalities based on different scenarios, thus highlighting a fundamental difference between personality in LLMs and humans. With our work, we call for a re-evaluation of personality definition and measurement in LLMs.