mlsanityData to 14 Oct 2026

The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models

arXiv 2507.16076 ↗First posted 21 Jul 2025cs.CLcited by 42

Not on the map. Why?

Abstract (arXiv)

Persona prompting is increasingly used in large language models (LLMs) to simulate views of various sociodemographic groups. However, how a persona prompt is formulated can significantly affect outcomes, raising concerns about the fidelity of such simulations. Using five open-source LLMs, we systematically examine how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence LLM simulations across 15 intersectional demographic groups in both open- and closed-ended tasks. Our findings show that LLMs struggle to simulate marginalized groups but that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment. Surprisingly, smaller models like OLMo-2-7B outperform larger ones such as Llama-3.3-70B. Our findings offer actionable guidance for designing sociodemographic persona prompts in LLM-based simulation studies.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

34 of 56 references.

References · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Whose Opinions Do Language Models Reflect?cited by 773
  4. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  5. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  6. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  7. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  8. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  9. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 249
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  11. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  12. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 199
22 more
  1. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  2. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  3. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
  4. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  5. 2025MMTEB: Massive Multilingual Text Embedding Benchmarkcited by 125
  6. 2023Aligning Language Models to User Opinionscited by 99
  7. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  8. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  9. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  10. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  11. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  12. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 51
  13. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  14. 2024Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Contextcited by 41
  15. 2024Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personascited by 37
  16. 2024Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humanscited by 31
  17. 2024Large Language Models Show Human-like Social Desirability Biases in Survey Responsescited by 28
  18. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 25
  19. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  20. 2024Stop! In the Name of Flaws: Disentangling Personal Names and Sociodemographic Attributes in NLPcited by 22
  21. 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
  22. 2023Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 32.

42 papers.

Sort:

By area:

  1. 2025The threat of analytic flexibility in using large language models to simulate human datacited by 15
  2. 2025Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Modelscited by 8
  3. 2026The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Modelscited by 8
  4. 2025German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studiescited by 5
  5. 2026Post-training makes large language models less human-likecited by 5
  6. 2026When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responsescited by 5
  7. 2026When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMscited by 4
  8. 2025Addressing Longstanding Challenges in Cognitive Science with Language Modelscited by 3
  9. 2025Dutch Metaphor Extraction from Cancer Patients' Interviews and Forum Data using LLMs and Human in the Loopcited by 3
  10. 2026Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interestcited by 3
  11. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?cited by 2
  12. 2026Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perceptioncited by 2
30 more
  1. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  2. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  3. 2026When Persona Attributes Improve Population Alignment in Large Language Modelscited by 2
  4. 2025A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian Historycited by 1
  5. 2025Prompt Fairness: Sub-group Disparities in LLMscited by 1
  6. 2026SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agentscited by 1
  7. 2026Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat: A Prominence-Stratified Cross-Provider Auditcited by 1
  8. 2026Inference-Time Steering for Cross-Lingual Factual Consistency in LLMscited by 1
  9. 2026Personalization, Personas, and Forecasting in Value Alignmentcited by 1
  10. 2026Role Steering of Language Models for Social Simulationscited by 1
  11. 2026When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutabilitycited by 1
  12. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  13. 2025Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novelscited by 0
  14. 2026How Value Induction Reshapes LLM Behaviourcited by 0
  15. 2026In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scorescited by 0
  16. 2026Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agentscited by 0
  17. 2026When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detectioncited by 0
  18. 2026Occupational Prompting Reveals Cultural Bias in Large Language Modelscited by 0
  19. 2026Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Modelscited by 0
  20. 2026Beyond Cultural Persona Bias: Response Geometry as a Measure for Language Models and Peoplecited by 0
  21. 2026The Librarian Who Refused to Code: Model-Dependent Identity Enactment in LLM Code Generationcited by 0
  22. 2026More Is Not More: What Matters for Diversity in LLM Opinions?cited by 0
  23. 2026Group Alignment-Induced Sycophancy: A Two-Sided Evaluation of Steerable Pluralistic Alignmentcited by 0
  24. 2026Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Studycited by 0
  25. 2026ExpertIVS: Sociological Expert Driven Individual Value Simulation in Large Language Modelscited by 0
  26. 2026LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurtscited by 0
  27. 2026The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysiscited by 0
  28. 2026Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Modelscited by 0
  29. 2026How Far Do Persona Effects Generalize in Language Models?cited by 0
  30. 2026Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulationcited by 0