mlsanityData to 14 Oct 2026

Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics

arXiv 2406.14703 ↗First posted 20 Jun 2024cs.CL cs.AIcited by 59

Not on the map. Why?

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have led to their adaptation in various domains as conversational agents. We wonder: can personality tests be applied to these agents to analyze their behavior, similar to humans? We introduce TRAIT, a new benchmark consisting of 8K multi-choice questions designed to assess the personality of LLMs. TRAIT is built on two psychometrically validated small human questionnaires, Big Five Inventory (BFI) and Short Dark Triad (SD-3), enhanced with the ATOMIC-10X knowledge graph to a variety of real-world scenarios. TRAIT also outperforms existing personality tests for LLMs in terms of reliability and validity, achieving the highest scores across four key metrics: Content Validity, Internal Validity, Refusal Rate, and Reliability. Using TRAIT, we reveal two notable insights into personalities of LLMs: 1) LLMs exhibit distinct and consistent personality, which is highly influenced by their training data (e.g., data used for alignment tuning), and 2) current prompting techniques have limited effectiveness in eliciting certain traits, such as high psychopathy or low conscientiousness, suggesting the need for further research in this direction.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 38 of its 108 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

38 of 108 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
26 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  2. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  3. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  4. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  5. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  6. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  7. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 179
  8. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  9. 2024Large Language Models are Geographically Biasedcited by 93
  10. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  11. 2023Humanoid Agents: Platform for Simulating Human-like Generative Agentscited by 73
  12. 2020TransOMCS: From Linguistic Graphs to Commonsense Knowledgecited by 52
  13. 2022Estimating the Personality of White-Box Language Modelscited by 48
  14. 2023CGMI: Configurable General Multi-Agent Interaction Frameworkcited by 42
  15. 2022Identifying and Manipulating the Personality Traits of Language Modelscited by 40
  16. 2023Self-Assessment Tests are Unreliable Measures of LLM Personalitycited by 39
  17. 2022Meet Your Favorite Character: Open-domain Chatbot Mimicking Fictional Characters with only a Few Utterancescited by 32
  18. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  19. 2022Weakly Supervised Data Augmentation Through Prompting for Dialogue Understandingcited by 32
  20. 2023Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMscited by 32
  21. 2022Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Qualitycited by 22
  22. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21
  23. 2022Measuring and Improving Semantic Diversity of Dialogue Generationcited by 16
  24. 2024Pearl: A Review-driven Persona-Knowledge Grounded Conversational Recommendation Datasetcited by 16
  25. 2022Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillationcited by 14
  26. 2023AI Text-to-Behavior: A Study In Steerabilitycited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 28.

59 papers.

Sort:

By area:

  1. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  2. 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation Historycited by 26
  3. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 24
  4. 2025Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Itemscited by 18
  5. 2025Scaffolding Empathy: Training Counselors with Simulated Patients and Utterance-level Performance Visualizationscited by 14
  6. 2025Bullying the Machine: How Personas Increase LLM Vulnerabilitycited by 9
  7. 2025Can LLMs Express Personality Across Cultures? Introducing CulturalPersonas for Evaluating Trait Alignmentcited by 9
  8. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  9. 2025A validity-guided workflow for robust large language model research in psychologycited by 8
  10. 2025A Review of Incorporating Psychological Theories in LLMscited by 6
  11. 2025Can LLMs Generate Behaviors for Embodied Virtual Agents Based on Personality Traits?cited by 5
  12. 2025Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Promptscited by 4
38 more
  1. 2025On the Adaptive Psychological Persuasion of Large Language Modelscited by 4
  2. 2025Scaling Personality Control in LLMs with Big Five Scaler Promptscited by 4
  3. 2026Persona Cartography: Charting Language Model Personality Traits in Weight Spacecited by 4
  4. 2025Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMscited by 3
  5. 2025Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediatorscited by 3
  6. 2025Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Needcited by 3
  7. 2025GRAVITY: A Framework for Personalized Text Generation via Profile-Grounded Synthetic Preferencescited by 3
  8. 2025LLMs Can Get "Brain Rot": A Pilot Study on Twitter/Xcited by 3
  9. 2025Profile-LLM: Dynamic Profile Optimization for Realistic Personality Expression in LLMscited by 3
  10. 2026AI-exhibited Personality Traits Can Shape Human Self-concept through Conversationscited by 3
  11. 2026Understanding Persuasive Interactions between Generative Social Agents and Humans: The Knowledge-based Persuasion Model (KPM)cited by 3
  12. 2025Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Realitycited by 2
  13. 2025Human Psychometric Questionnaires Mischaracterize LLM Behaviorcited by 2
  14. 2025Quantifying Data Contamination in Psychometric Evaluations of LLMscited by 2
  15. 2026Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Studycited by 2
  16. 2026ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?cited by 2
  17. 2025Persona Dynamics: Unveiling the Impact of Personality Traits on Agents in Text-Based Gamescited by 1
  18. 2025IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimizationcited by 1
  19. 2025Decoding Emergent Big Five Traits in Large Language Models: Temperature-Dependent Expression and Architectural Clusteringcited by 1
  20. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
  21. 2026Persuasion Propagation: Does Persuasion Change AI Agent Behavior?cited by 1
  22. 2024Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Rolescited by 0
  23. 2025Images Amplify Misinformation Sharing in Vision-Language Modelscited by 0
  24. 2025The Narcissus Hypothesis: Descending to the Rung of Illusioncited by 0
  25. 2025Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Testscited by 0
  26. 2025MindShift: Analyzing Language Models' Reactions to Psychological Promptscited by 0
  27. 2026Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoderscited by 0
  28. 2026The Company You Keep: How LLMs Respond to Dark Triad Traitscited by 0
  29. 2026"Dark Triad" Model Organisms of Misalignment: Narrow Fine-Tuning Mirrors Human Antisocial Behaviorcited by 0
  30. 2026Is AI Catching Up to Human Expression? Exploring Emotion, Personality, Authorship, and Linguistic Style in English and Arabic with Six Large Language Modelscited by 0
  31. 2026The Effects of Visual Priming on Cooperative Behavior in Vision-Language Modelscited by 0
  32. 2026The Shared Latent Structure of LLM Psychometric Responsescited by 0
  33. 2026Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMscited by 0
  34. 2026The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometricscited by 0
  35. 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMscited by 0
  36. 2026AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Modelscited by 0
  37. 2026Mechanistic Personality Analysis of LLMs: Steering Personality via Latent Feature Interventionscited by 0
  38. 2026Beyond Cultural Persona Bias: Response Geometry as a Measure for Language Models and Peoplecited by 0

50 most cited of 59 All 59 (JSON)