# Quantifying the Persona Effect in LLM Simulations

arXiv: https://arxiv.org/abs/2402.10811 · HTML: https://mlsanity.com/paper/2402.10811/

## Key facts

- Cited by 169 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 69 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [Designing Toxic Content Classification for a Diversity of Perspectives](https://mlsanity.com/paper/2106.04511/) (2021)
- [When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset](https://mlsanity.com/paper/2306.06826/) (2023)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [Incorporating Worker Perspectives into MTurk Annotation Practices for NLP](https://mlsanity.com/paper/2311.02802/) (2023)
- [How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets](https://mlsanity.com/paper/2309.01288/) (2023)

## Built on by (most cited first, 30 of 169)

- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?](https://mlsanity.com/paper/2406.13605/) (2024)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Virtual Personas for Language Models via an Anthology of Backstories](https://mlsanity.com/paper/2407.06576/) (2024)
- [Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents](https://mlsanity.com/paper/2402.12327/) (2024)
- [The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models](https://mlsanity.com/paper/2507.16076/) (2025)
- [Conformity in Large Language Models](https://mlsanity.com/paper/2410.12428/) (2024)
- [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025)
- [LLM-based Human Simulations Have Not Yet Been Reliable](https://mlsanity.com/paper/2501.08579/) (2025)
- [Value Profiles for Encoding Human Variation](https://mlsanity.com/paper/2503.15484/) (2025)
- [SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs](https://mlsanity.com/paper/2506.05598/) (2025)
- [LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Education](https://mlsanity.com/paper/2410.14012/) (2024)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas](https://mlsanity.com/paper/2406.14462/) (2024)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [Can Language Models Recognize Convincing Arguments?](https://mlsanity.com/paper/2404.00750/) (2024)
- [Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior](https://mlsanity.com/paper/2407.02099/) (2024)
- [The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models](https://mlsanity.com/paper/2406.11096/) (2024)
- [Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamics](https://mlsanity.com/paper/2506.12657/) (2025)
- [Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Items](https://mlsanity.com/paper/2505.01015/) (2025)
- [Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification](https://mlsanity.com/paper/2510.11408/) (2025)
- [Learning Personalized Agents from Human Feedback](https://mlsanity.com/paper/2602.16173/) (2026)
- [A Survey on LLM-based Conversational User Simulation](https://mlsanity.com/paper/2604.24977/) (2026)
- [Unmasking Conversational Bias in AI Multiagent Systems](https://mlsanity.com/paper/2501.14844/) (2025)
- [Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles](https://mlsanity.com/paper/2507.22168/) (2025)
- [GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy](https://mlsanity.com/paper/2407.18008/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables-demographic, social, and behavioral factors-impacts LLMs' ability to simulate diverse perspectives. We find that persona variables account for <10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating persona variables via prompting in LLMs provides modest but statistically significant improvements. Persona prompting is most effective in samples where many annotators disagree, but their disagreements are relatively minor. Notably, we find a linear relationship in our setting: the stronger the correlation between persona variables and human annotations, the more accurate the LLM predictions are using persona prompting. In a zero-shot setting, a powerful 70b model with persona prompting captures 81% of the annotation variance achievable by linear regression trained on ground truth annotations. However, for most subjective NLP datasets, where persona variables have limited explanatory power, the benefits of persona prompting are limited.