# Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation

arXiv: https://arxiv.org/abs/2306.01183 · HTML: https://mlsanity.com/paper/2306.01183/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 57 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities](https://mlsanity.com/paper/2301.04408/) (2023)
- [Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality](https://mlsanity.com/paper/2105.03484/) (2021)
- [MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection](https://mlsanity.com/paper/2109.08113/) (2021)
- [Pushing on Personality Detection from Verbal Behavior: A Transformer Meets Text Contours of Psycholinguistic Features](https://mlsanity.com/paper/2204.04629/) (2022)
- [Latent Human Traits in the Language of Social Media: An Open-Vocabulary Approach](https://mlsanity.com/paper/1705.08038/) (2017)
- [Team Phoenix at WASSA 2021: Emotion Analysis on News Stories with Pre-Trained Language Models](https://mlsanity.com/paper/2103.06057/) (2021)
- [Using Paraphrases to Study Properties of Contextual Embeddings](https://mlsanity.com/paper/2207.05553/) (2022)
- [Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment](https://mlsanity.com/paper/2112.13795/) (2021)

## Built on by (most cited first, 29 of 29)

- [You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments](https://mlsanity.com/paper/2311.09718/) (2023)
- [Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas](https://mlsanity.com/paper/2406.14462/) (2024)
- [PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection](https://mlsanity.com/paper/2310.20256/) (2023)
- [Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models](https://mlsanity.com/paper/2401.07115/) (2024)
- [ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models](https://mlsanity.com/paper/2406.04214/) (2024)
- [Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models](https://mlsanity.com/paper/2409.12106/) (2024)
- [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024)
- [When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions](https://mlsanity.com/paper/2507.20439/) (2025)
- [SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks](https://mlsanity.com/paper/2402.01980/) (2024)
- [Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization](https://mlsanity.com/paper/2311.09510/) (2023)
- [Generative Psycho-Lexical Approach for Constructing Value Systems in Large Language Models](https://mlsanity.com/paper/2502.02444/) (2025)
- [From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP](https://mlsanity.com/paper/2601.07988/) (2026)
- [Illuminating the Black Box: A Psychometric Investigation into the Multifaceted Nature of Large Language Models](https://mlsanity.com/paper/2312.14202/) (2023)
- [Extroversion or Introversion? Controlling The Personality of Your Large Language Models](https://mlsanity.com/paper/2406.04583/) (2024)
- [Twenty Years of Personality Computing: Threats, Challenges and Future Directions](https://mlsanity.com/paper/2503.02082/) (2025)
- [Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Models](https://mlsanity.com/paper/2512.06991/) (2025)
- [Context-Value-Action Architecture for Value-Driven Large Language Model Agents](https://mlsanity.com/paper/2604.05939/) (2026)
- [Humanity in AI: Detecting the Personality of Large Language Models](https://mlsanity.com/paper/2410.08545/) (2024)
- [Psychologically Enhanced AI Agents](https://mlsanity.com/paper/2509.04343/) (2025)
- [When AI Takes the Couch: Psychometric Jailbreaks Reveal Internal Conflict in Frontier Models](https://mlsanity.com/paper/2512.04124/) (2025)
- [LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion](https://mlsanity.com/paper/2606.29900/) (2026)
- [Personality Prediction from Life Stories using Language Models](https://mlsanity.com/paper/2506.19258/) (2025)
- [Mind Reading or Misreading? LLMs on the Big Five Personality Test](https://mlsanity.com/paper/2511.23101/) (2025)
- [A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategies](https://mlsanity.com/paper/2602.06015/) (2026)
- [Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmark](https://mlsanity.com/paper/2603.17838/) (2026)
- [Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning](https://mlsanity.com/paper/2603.27057/) (2026)
- [HyperPersona: A Multi-Level Hypergraph Framework for Text-Based Automatic Personality Prediction](https://mlsanity.com/paper/2605.17355/) (2026)
- [When Personality Meets Quantization: A Layer-wise MBTI Analysis of Quantized LLMs](https://mlsanity.com/paper/2608.25977/) (2026)
- [Cross-Protocol Personality Assessment with LLMs: Aligning Behavioral Evidence with Human Ratings](https://mlsanity.com/paper/2610.16050/) (2026)

## Abstract (arXiv)

Very large language models (LLMs) perform extremely well on a spectrum of NLP tasks in a zero-shot setting. However, little is known about their performance on human-level NLP problems which rely on understanding psychological concepts, such as assessing personality traits. In this work, we investigate the zero-shot ability of GPT-3 to estimate the Big 5 personality traits from users' social media posts. Through a set of systematic experiments, we find that zero-shot GPT-3 performance is somewhat close to an existing pre-trained SotA for broad classification upon injecting knowledge about the trait in the prompts. However, when prompted to provide fine-grained classification, its performance drops to close to a simple most frequent class (MFC) baseline. We further analyze where GPT-3 performs better, as well as worse, than a pretrained lexical model, illustrating systematic errors that suggest ways to improve LLMs on human-level NLP tasks.