# Large Language Models Show Human-like Social Desirability Biases in Survey Responses

arXiv: https://arxiv.org/abs/2405.06058 · HTML: https://mlsanity.com/paper/2405.06058/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 6 of 6)

- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)

## Built on by (most cited first, 28 of 28)

- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models](https://mlsanity.com/paper/2507.16076/) (2025)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [DHP Benchmark: Are LLMs Good NLG Evaluators?](https://mlsanity.com/paper/2408.13704/) (2024)
- [LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings](https://mlsanity.com/paper/2510.08338/) (2025)
- [Self-assessment, Exhibition, and Recognition: a Review of Personality in Large Language Models](https://mlsanity.com/paper/2406.17624/) (2024)
- [The Bias is in the Details: An Assessment of Cognitive Bias in LLMs](https://mlsanity.com/paper/2509.22856/) (2025)
- [Bullying the Machine: How Personas Increase LLM Vulnerability](https://mlsanity.com/paper/2505.12692/) (2025)
- [Be.FM: Open Foundation Models for Human Behavior](https://mlsanity.com/paper/2505.23058/) (2025)
- [AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation](https://mlsanity.com/paper/2601.00930/) (2026)
- [Scaling Law in LLM Simulated Personality: More Detailed and Realistic Persona Profile Is All You Need](https://mlsanity.com/paper/2510.11734/) (2025)
- [Debunking with Dialogue? Exploring AI-Generated Counterspeech to Challenge Conspiracy Theories](https://mlsanity.com/paper/2504.16604/) (2025)
- [A Survey on Training-free Alignment of Large Language Models](https://mlsanity.com/paper/2508.09016/) (2025)
- [Language Models Identify Ambiguities and Exploit Loopholes](https://mlsanity.com/paper/2508.19546/) (2025)
- [Human Psychometric Questionnaires Mischaracterize LLM Behavior](https://mlsanity.com/paper/2509.10078/) (2025)
- [Algorithmic Fairness in NLP: Persona-Infused LLMs for Human-Centric Hate Speech Detection](https://mlsanity.com/paper/2510.19331/) (2025)
- [IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimization](https://mlsanity.com/paper/2508.08719/) (2025)
- [RBCorr: Response Bias Correction in Language Models](https://mlsanity.com/paper/2602.12445/) (2026)
- [BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks](https://mlsanity.com/paper/2606.24162/) (2026)
- [Automated Item Neutralization for Non-Cognitive Scales: A Large Language Model Approach to Reducing Social-Desirability Bias](https://mlsanity.com/paper/2509.19314/) (2025)
- [Who are you, ChatGPT? Personality and Demographic Style in LLM-Generated Content](https://mlsanity.com/paper/2510.11434/) (2025)
- [Status Hierarchies in Language Models](https://mlsanity.com/paper/2601.17577/) (2026)
- [Projective Psychological Assessment of Large Multimodal Models Using Thematic Apperception Tests](https://mlsanity.com/paper/2602.17108/) (2026)
- [The Shared Latent Structure of LLM Psychometric Responses](https://mlsanity.com/paper/2605.05080/) (2026)
- [The Granularity Axis: A Micro-to-Macro Latent Direction for Social Roles in Language Models](https://mlsanity.com/paper/2605.06196/) (2026)
- [Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?](https://mlsanity.com/paper/2605.16996/) (2026)
- [Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?](https://mlsanity.com/paper/2605.17694/) (2026)
- [I Understand How You Feel: Enhancing Deeper Emotional Support Through Multilingual Emotional Validation in Dialogue System](https://mlsanity.com/paper/2606.11875/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) become widely used to model and simulate human behavior, understanding their biases becomes critical. We developed an experimental framework using Big Five personality surveys and uncovered a previously undetected social desirability bias in a wide range of LLMs. By systematically varying the number of questions LLMs were exposed to, we demonstrate their ability to infer when they are being evaluated. When personality evaluation is inferred, LLMs skew their scores towards the desirable ends of trait dimensions (i.e., increased extraversion, decreased neuroticism, etc). This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent models, with GPT-4's survey responses changing by 1.20 (human) standard deviations and Llama 3's by 0.98 standard deviations-very large effects. This bias is robust to randomization of question order and paraphrasing. Reverse-coding all the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias. Our findings reveal an emergent social desirability bias and suggest constraints on profiling LLMs with psychometric tests and on using LLMs as proxies for human participants.