mlsanityData to 14 Oct 2026

Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

arXiv 2306.04308 ↗First posted 7 Jun 2023cs.AI cs.CL cs.HCcited by 16

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 11 of its 63 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

11 of 63 references.

References · 11

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2022Using cognitive psychology to understand GPT-3cited by 398
  6. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  7. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  8. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  9. 2023The Self-Perception and Political Biases of ChatGPTcited by 66
  10. 2023Revisiting the Reliability of Psychological Scales on Large Language Modelscited by 16
  11. 2022Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turkcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  2. 2026PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebracited by 12
  3. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applicationscited by 10
  4. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  5. 2025Evaluating LLM Safety Across Child Development Stages: A Simulated Agent Approachcited by 3
  6. 2025The Effect of Belief Boxes and Open-mindedness on Persuasioncited by 3
  7. 2026Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Eventscited by 2
  8. 2025Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasmcited by 1
  9. 2025Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysiscited by 1
  10. 2025Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attackscited by 1
  11. 2025When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Modelscited by 1
  12. 2026LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humanscited by 1
4 more
  1. 2025The Narcissus Hypothesis: Descending to the Rung of Illusioncited by 0
  2. 2026The Shared Latent Structure of LLM Psychometric Responsescited by 0
  3. 2026Faking Good and Faking Bad in LLMs: Response Distortion Across Dark Triad Personality Traitscited by 0
  4. 2026Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Contentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History
  2. 2022Evaluating Psychological Safety of Large Language Models
  3. 2025Evaluating LLM Alignment on Personality Inference from Real-World Interview Data
  4. 2025In Silico Development of Psychometric Scales: Feasibility of Representative Population Data Simulation with LLMs
  5. 2023Do LLMs exhibit human-like response biases? A case study in survey design
  6. 2022Fine-tuning language models to find agreement among humans with diverse preferences
  7. 2025Gender and Positional Biases in LLM-Based Hiring Decisions: Evidence from Comparative CV/R'esum'e Evaluations
  8. 2024Personality Alignment of Large Language Models