mlsanityData to 7 Oct 2026

Quantifying the Persona Effect in LLM Simulations

arXiv 2402.10811 ↗First posted 16 Feb 2024cs.CL cs.CYcited by 164

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables-demographic, social, and behavioral factors-impacts LLMs' ability to simulate diverse perspectives. We find that persona variables account for <10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating persona variables via prompting in LLMs provides modest but statistically significant improvements. Persona prompting is most effective in samples where many annotators disagree, but their disagreements are relatively minor. Notably, we find a linear relationship in our setting: the stronger the correlation between persona variables and human annotations, the more accurate the LLM predictions are using persona prompting. In a zero-shot setting, a powerful 70b model with persona prompting captures 81% of the annotation variance achievable by linear regression trained on ground truth annotations. However, for most subjective NLP datasets, where persona variables have limited explanatory power, the benefits of persona prompting are limited.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Builds on 26 of its 69 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

26 of 69 references.

References · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023Whose Opinions Do Language Models Reflect?cited by 759
  5. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  6. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  7. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  8. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  9. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  10. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  11. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  12. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
14 more
  1. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  2. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 129
  3. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  4. 2024Systematic Biases in LLM Simulations of Debatescited by 98
  5. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  6. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  7. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  8. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  9. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  10. 2021Designing Toxic Content Classification for a Diversity of Perspectivescited by 63
  11. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 51
  12. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  13. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  14. 2023How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweetscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 67; 2026: 74.

164 papers.

Sort:

By area:

  1. 2024Can LLM be a Personalized Judge?cited by 99
  2. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  3. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  4. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 62
  5. 2025Flipping the Dialogue: Training and Evaluating User Language Modelscited by 50
  6. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  7. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  8. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agentscited by 48
  9. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  10. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 40
  11. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 31
  12. 2024Conformity in Large Language Modelscited by 30
38 more
  1. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  2. 2025Value Profiles for Encoding Human Variationcited by 29
  3. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  4. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  5. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  6. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  7. 2024Can Language Models Recognize Convincing Arguments?cited by 21
  8. 2024Helpful assistant or fruitful facilitator? Investigating how personas affect language model behaviorcited by 21
  9. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  10. 2025Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamicscited by 19
  11. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 19
  12. 2025Value Portrait: Assessing Language Models' Values through Psychometrically and Ecologically Valid Itemscited by 18
  13. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 17
  14. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 16
  15. 2026Learning Personalized Agents from Human Feedbackcited by 16
  16. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
  17. 2024GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancycited by 14
  18. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparencycited by 14
  19. 2026A Survey on LLM-based Conversational User Simulationcited by 14
  20. 2025Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detectioncited by 13
  21. 2026Exploring Silicon-Based Societies: An Early Study of the Moltbook Agent Communitycited by 13
  22. 2025PersonaAgent: Bridging Memory and Action for Personalized LLM Agentscited by 11
  23. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  24. 2025Enhancing Jailbreak Attacks on LLMs via Persona Promptscited by 10
  25. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  26. 2026Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLPcited by 10
  27. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  28. 2025Improving Language Model Personas via Rationalization with Psychological Scaffoldscited by 9
  29. 2024Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recoverycited by 8
  30. 2025Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generationcited by 8
  31. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 8
  32. 2024Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensivenesscited by 7
  33. 2024Rethinking Emotion Annotations in the Era of Large Language Modelscited by 7
  34. 2025Aligning LLM with human travel choices: a persona-based embedding learning approachcited by 7
  35. 2026Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondentscited by 7
  36. 2024Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Developmentcited by 6
  37. 2025HARBOR: Exploring Persona Dynamics in Multi-Agent Competitioncited by 6
  38. 2025Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essayscited by 6

50 most cited of 164 All 164 (JSON)