# Can LLMs Infer Personality from Real World Conversations?

arXiv: https://arxiv.org/abs/2507.14355 · HTML: https://mlsanity.com/paper/2507.14355/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 37 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Generative AI and Its Impact on Personalized Intelligent Tutoring Systems](https://mlsanity.com/paper/2410.10650/) (2024)
- [Predicting the Big Five Personality Traits in Chinese Counselling Dialogues Using Large Language Models](https://mlsanity.com/paper/2406.17287/) (2024)
- [Automatic Extraction of Personality from Text: Challenges and Opportunities](https://mlsanity.com/paper/1910.09916/) (2019)
- [Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction](https://mlsanity.com/paper/2304.00377/) (2023)
- [Detecting PTSD in Clinical Interviews: A Comparative Analysis of NLP Methods and Large Language Models](https://mlsanity.com/paper/2504.01216/) (2025)
- [Emotions, Context, and Substance Use in Adolescents: A Large Language Model Analysis of Reddit Posts](https://mlsanity.com/paper/2501.14037/) (2025)
- [Personality Detection and Analysis using Twitter Data](https://mlsanity.com/paper/2309.05497/) (2023)

## Built on by (most cited first, 4 of 4)

- [Evaluating LLM Alignment on Personality Inference from Real-World Interview Data](https://mlsanity.com/paper/2509.13244/) (2025)
- [AI-exhibited Personality Traits Can Shape Human Self-concept through Conversations](https://mlsanity.com/paper/2601.12727/) (2026)
- [Beyond Asking: A Pipeline for Personalized Game Generation that Reads Players from Behavior](https://mlsanity.com/paper/2608.16196/) (2026)
- [Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?](https://mlsanity.com/paper/2603.27694/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) such as OpenAI's GPT-4 and Meta's LLaMA offer a promising approach for scalable personality assessment from open-ended language. However, inferring personality traits remains challenging, and earlier work often relied on synthetic data or social media text lacking psychometric validity. We introduce a real-world benchmark of 555 semi-structured interviews with BFI-10 self-report scores for evaluating LLM-based personality inference. Three state-of-the-art LLMs (GPT-4.1 Mini, Meta-LLaMA, and DeepSeek) were tested using zero-shot prompting for BFI-10 item prediction and both zero-shot and chain-of-thought prompting for Big Five trait inference. All models showed high test-retest reliability, but construct validity was limited: correlations with ground-truth scores were weak (max Pearson's r = 0.27), interrater agreement was low (Cohen's kappa < 0.10), and predictions were biased toward moderate or high trait levels. Chain-of-thought prompting and longer input context modestly improved distributional alignment, but not trait-level accuracy. These results underscore limitations in current LLM-based personality inference and highlight the need for evidence-based development for psychological applications.