# Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics

arXiv: https://arxiv.org/abs/2406.14703 · HTML: https://mlsanity.com/paper/2406.14703/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 108 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [Humanoid Agents: Platform for Simulating Human-like Generative Agents](https://mlsanity.com/paper/2310.05418/) (2023)
- [TransOMCS: From Linguistic Graphs to Commonsense Knowledge](https://mlsanity.com/paper/2005.00206/) (2020)
- [Estimating the Personality of White-Box Language Models](https://mlsanity.com/paper/2204.12000/) (2022)
- [CGMI: Configurable General Multi-Agent Interaction Framework](https://mlsanity.com/paper/2308.12503/) (2023)
- [Identifying and Manipulating the Personality Traits of Language Models](https://mlsanity.com/paper/2212.10276/) (2022)
- [Self-Assessment Tests are Unreliable Measures of LLM Personality](https://mlsanity.com/paper/2309.08163/) (2023)
- [Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterances](https://mlsanity.com/paper/2204.10825/) (2022)
- [Aligning to Social Norms and Values in Interactive Narratives](https://mlsanity.com/paper/2205.01975/) (2022)

## Built on by (most cited first, 30 of 59)

- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History](https://mlsanity.com/paper/2508.04826/) (2025)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items](https://mlsanity.com/paper/2505.01015/) (2025)
- [Scaffolding Empathy: Training Counselors with Simulated Patients and Utterance-level Performance Visualizations](https://mlsanity.com/paper/2502.18673/) (2025)
- [Bullying the Machine: How Personas Increase LLM Vulnerability](https://mlsanity.com/paper/2505.12692/) (2025)
- [Can LLMs Express Personality Across Cultures? Introducing CulturalPersonas for Evaluating Trait Alignment](https://mlsanity.com/paper/2506.05670/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Can LLMs Generate Behaviors for Embodied Virtual Agents Based on Personality Traits?](https://mlsanity.com/paper/2508.21087/) (2025)
- [Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts](https://mlsanity.com/paper/2504.21117/) (2025)
- [On the Adaptive Psychological Persuasion of Large Language Models](https://mlsanity.com/paper/2506.06800/) (2025)
- [Scaling Personality Control in LLMs with Big Five Scaler Prompts](https://mlsanity.com/paper/2508.06149/) (2025)
- [Persona Cartography: Charting Language Model Personality Traits in Weight Space](https://mlsanity.com/paper/2607.07916/) (2026)
- [Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs](https://mlsanity.com/paper/2506.02659/) (2025)
- [Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators](https://mlsanity.com/paper/2507.05890/) (2025)
- [Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need](https://mlsanity.com/paper/2510.11734/) (2025)
- [GRAVITY: A Framework for Personalized Text Generation via Profile-Grounded Synthetic Preferences](https://mlsanity.com/paper/2510.11952/) (2025)
- [LLMs Can Get "Brain Rot": A Pilot Study on Twitter/X](https://mlsanity.com/paper/2510.13928/) (2025)
- [Profile-LLM: Dynamic Profile Optimization for Realistic Personality Expression in LLMs](https://mlsanity.com/paper/2511.19852/) (2025)
- [AI-exhibited Personality Traits Can Shape Human Self-concept through Conversations](https://mlsanity.com/paper/2601.12727/) (2026)
- [Understanding Persuasive Interactions between Generative Social Agents and Humans: The Knowledge-based Persuasion Model (KPM)](https://mlsanity.com/paper/2602.11483/) (2026)
- [Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Reality](https://mlsanity.com/paper/2503.16457/) (2025)
- [Human Psychometric Questionnaires Mischaracterize LLM Behavior](https://mlsanity.com/paper/2509.10078/) (2025)
- [Quantifying Data Contamination in Psychometric Evaluations of LLMs](https://mlsanity.com/paper/2510.07175/) (2025)
- [Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study](https://mlsanity.com/paper/2602.17262/) (2026)
- [ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?](https://mlsanity.com/paper/2606.05553/) (2026)
- [Persona Dynamics: Unveiling the Impact of Personality Traits on Agents in Text-Based Games](https://mlsanity.com/paper/2504.06868/) (2025)
- [IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization](https://mlsanity.com/paper/2508.08719/) (2025)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have led to their adaptation in various domains as conversational agents. We wonder: can personality tests be applied to these agents to analyze their behavior, similar to humans? We introduce TRAIT, a new benchmark consisting of 8K multi-choice questions designed to assess the personality of LLMs. TRAIT is built on two psychometrically validated small human questionnaires, Big Five Inventory (BFI) and Short Dark Triad (SD-3), enhanced with the ATOMIC-10X knowledge graph to a variety of real-world scenarios. TRAIT also outperforms existing personality tests for LLMs in terms of reliability and validity, achieving the highest scores across four key metrics: Content Validity, Internal Validity, Refusal Rate, and Reliability. Using TRAIT, we reveal two notable insights into personalities of LLMs: 1) LLMs exhibit distinct and consistent personality, which is highly influenced by their training data (e.g., data used for alignment tuning), and 2) current prompting techniques have limited effectiveness in eliciting certain traits, such as high psychopathy or low conscientiousness, suggesting the need for further research in this direction.