# Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench

arXiv: https://arxiv.org/abs/2310.01386 · HTML: https://mlsanity.com/paper/2310.01386/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 92 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [Evaluating the Moral Beliefs Encoded in LLMs](https://mlsanity.com/paper/2307.14324/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Evaluating and Inducing Personality in Pre-trained Language Models](https://mlsanity.com/paper/2206.07550/) (2022)
- [Who is GPT-3? An Exploration of Personality, Values and Demographics](https://mlsanity.com/paper/2209.14338/) (2022)
- [Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study](https://mlsanity.com/paper/2304.04339/) (2023)
- [CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?](https://mlsanity.com/paper/2306.16636/) (2023)
- [ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark](https://mlsanity.com/paper/2303.13648/) (2023)
- [The Self-Perception and Political Biases of ChatGPT](https://mlsanity.com/paper/2304.07333/) (2023)
- [Emotional Intelligence of Large Language Models](https://mlsanity.com/paper/2307.09042/) (2023)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Estimating the Personality of White-Box Language Models](https://mlsanity.com/paper/2204.12000/) (2022)
- [Exploring the psychology of LLMs' Moral and Legal Reasoning](https://mlsanity.com/paper/2308.01264/) (2023)
- [How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?](https://mlsanity.com/paper/2306.01248/) (2023)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [Revisiting the Reliability of Psychological Scales on Large Language Models](https://mlsanity.com/paper/2305.19926/) (2023)
- [Exploring ChatGPT's Empathic Abilities](https://mlsanity.com/paper/2308.03527/) (2023)
- [Is GPT a Computational Model of Emotion? Detailed Analysis](https://mlsanity.com/paper/2307.13779/) (2023)
- [ChatGPT Is More Likely to Be Perceived as Male Than Female](https://mlsanity.com/paper/2305.12564/) (2023)

## Built on by (most cited first, 28 of 28)

- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety](https://mlsanity.com/paper/2401.11880/) (2024)
- [metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models](https://mlsanity.com/paper/2407.12844/) (2024)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis](https://mlsanity.com/paper/2405.07248/) (2024)
- [Challenging the Validity of Personality Tests for Large Language Models](https://mlsanity.com/paper/2311.05297/) (2023)
- [The Better Angels of Machine Personality: How Personality Relates to LLM Safety](https://mlsanity.com/paper/2407.12344/) (2024)
- [Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models](https://mlsanity.com/paper/2401.07115/) (2024)
- [Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Models](https://mlsanity.com/paper/2410.11459/) (2024)
- [Can LLMs make trade-offs involving stipulated pain and pleasure states?](https://mlsanity.com/paper/2411.02432/) (2024)
- [PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebra](https://mlsanity.com/paper/2602.15669/) (2026)
- [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024)
- [Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Design](https://mlsanity.com/paper/2508.17573/) (2025)
- [Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications](https://mlsanity.com/paper/2505.00049/) (2025)
- [Is Self-knowledge and Action Consistent or Not: Investigating Large Language Model's Personality](https://mlsanity.com/paper/2402.14679/) (2024)
- [Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales](https://mlsanity.com/paper/2409.19655/) (2024)
- [Evaluating Alignment of Behavioral Dispositions in LLMs](https://mlsanity.com/paper/2602.11328/) (2026)
- [CogGPT: Unleashing the Power of Cognitive Dynamics on Large Language Models](https://mlsanity.com/paper/2401.08438/) (2024)
- [AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behavior](https://mlsanity.com/paper/2403.13433/) (2024)
- [Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective](https://mlsanity.com/paper/2406.14023/) (2024)
- [An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios](https://mlsanity.com/paper/2402.01723/) (2024)
- [Exploring Big Five Personality and AI Capability Effects in LLM-Simulated Negotiation Dialogues](https://mlsanity.com/paper/2506.15928/) (2025)
- [TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health](https://mlsanity.com/paper/2603.03047/) (2026)
- [Consistency Matters: Explore LLMs Consistency From a Black-Box Perspective](https://mlsanity.com/paper/2402.17411/) (2024)
- [SeSaMe: A Framework to Simulate Self-Reported Ground Truth for Mental Health Sensing Studies](https://mlsanity.com/paper/2403.17219/) (2024)
- [Enhancing AI-Driven Psychological Consultation: Layered Prompts with Large Language Models](https://mlsanity.com/paper/2408.16276/) (2024)
- [Leveraging Interview-Informed LLMs to Model Survey Responses: Comparative Insights from AI-Generated and Human Data](https://mlsanity.com/paper/2505.21997/) (2025)
- [When Personality Meets Quantization: A Layer-wise MBTI Analysis of Quantized LLMs](https://mlsanity.com/paper/2608.25977/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have recently showcased their remarkable capacities, not only in natural language processing tasks but also across diverse domains such as clinical medicine, legal consultation, and education. LLMs become more than mere applications, evolving into assistants capable of addressing diverse user requests. This narrows the distinction between human beings and artificial intelligence agents, raising intriguing questions regarding the potential manifestation of personalities, temperaments, and emotions within LLMs. In this paper, we propose a framework, PsychoBench, for evaluating diverse psychological aspects of LLMs. Comprising thirteen scales commonly used in clinical psychology, PsychoBench further classifies these scales into four distinct categories: personality traits, interpersonal relationships, motivational tests, and emotional abilities. Our study examines five popular models, namely text-davinci-003, gpt-3.5-turbo, gpt-4, LLaMA-2-7b, and LLaMA-2-13b. Additionally, we employ a jailbreak approach to bypass the safety alignment protocols and test the intrinsic natures of LLMs. We have made PsychoBench openly accessible via https://github.com/CUHK-ARISE/PsychoBench.