# Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

arXiv: https://arxiv.org/abs/2306.04308 · HTML: https://mlsanity.com/paper/2306.04308/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 63 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 11 of 11)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [The Self-Perception and Political Biases of ChatGPT](https://mlsanity.com/paper/2304.07333/) (2023)
- [Revisiting the Reliability of Psychological Scales on Large Language Models](https://mlsanity.com/paper/2305.19926/) (2023)
- [Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk](https://mlsanity.com/paper/2209.11344/) (2022)

## Built on by (most cited first, 16 of 16)

- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebra](https://mlsanity.com/paper/2602.15669/) (2026)
- [Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications](https://mlsanity.com/paper/2505.00049/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approach](https://mlsanity.com/paper/2510.05484/) (2025)
- [The Effect of Belief Boxes and Open-mindedness on Persuasion](https://mlsanity.com/paper/2512.06573/) (2025)
- [Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events](https://mlsanity.com/paper/2608.06485/) (2026)
- [Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm](https://mlsanity.com/paper/2501.02532/) (2025)
- [Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis](https://mlsanity.com/paper/2506.12189/) (2025)
- [Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks](https://mlsanity.com/paper/2510.14207/) (2025)
- [When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Models](https://mlsanity.com/paper/2512.04124/) (2025)
- [LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans](https://mlsanity.com/paper/2604.19787/) (2026)
- [The Narcissus Hypothesis: Descending to the Rung of Illusion](https://mlsanity.com/paper/2509.17999/) (2025)
- [The Shared Latent Structure of LLM Psychometric Responses](https://mlsanity.com/paper/2605.05080/) (2026)
- [Faking Good and Faking Bad in LLMs: Response Distortion Across Dark Triad Personality Traits](https://mlsanity.com/paper/2609.17534/) (2026)
- [Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Content](https://mlsanity.com/paper/2609.30563/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.