# The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models

arXiv: https://arxiv.org/abs/2507.16076 · HTML: https://mlsanity.com/paper/2507.16076/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 30 of 34)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models](https://mlsanity.com/paper/2402.16786/) (2024)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering](https://mlsanity.com/paper/2309.17249/) (2023)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions](https://mlsanity.com/paper/2502.16761/) (2025)
- [Virtual Personas for Language Models via an Anthology of Backstories](https://mlsanity.com/paper/2407.06576/) (2024)
- [You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments](https://mlsanity.com/paper/2311.09718/) (2023)
- [Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context](https://mlsanity.com/paper/2406.05972/) (2024)
- [Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas](https://mlsanity.com/paper/2408.06929/) (2024)
- [Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans](https://mlsanity.com/paper/2401.08495/) (2024)
- [Large Language Models Show Human-like Social Desirability Biases in Survey Responses](https://mlsanity.com/paper/2405.06058/) (2024)
- ["They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations](https://mlsanity.com/paper/2405.05378/) (2024)

## Built on by (most cited first, 30 of 42)

- [The threat of analytic flexibility in using large language models to simulate human data](https://mlsanity.com/paper/2509.13397/) (2025)
- [Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models](https://mlsanity.com/paper/2510.11586/) (2025)
- [The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models](https://mlsanity.com/paper/2604.24698/) (2026)
- [German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies](https://mlsanity.com/paper/2511.21722/) (2025)
- [Post-training makes large language models less human-like](https://mlsanity.com/paper/2605.07632/) (2026)
- [When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses](https://mlsanity.com/paper/2607.26348/) (2026)
- [When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs](https://mlsanity.com/paper/2605.29420/) (2026)
- [Addressing Longstanding Challenges in Cognitive Science with Language Models](https://mlsanity.com/paper/2511.00206/) (2025)
- [Dutch Metaphor Extraction from Cancer Patients' Interviews and Forum Data using LLMs and Human in the Loop](https://mlsanity.com/paper/2511.06427/) (2025)
- [Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest](https://mlsanity.com/paper/2604.08525/) (2026)
- [Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?](https://mlsanity.com/paper/2511.21218/) (2025)
- [Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception](https://mlsanity.com/paper/2604.28048/) (2026)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models](https://mlsanity.com/paper/2607.15277/) (2026)
- [When Persona Attributes Improve Population Alignment in Large Language Models](https://mlsanity.com/paper/2609.02526/) (2026)
- [A Cross-Lingual Analysis of Bias in Large Language Models Using Romanian History](https://mlsanity.com/paper/2510.02362/) (2025)
- [Prompt Fairness: Sub-group Disparities in LLMs](https://mlsanity.com/paper/2511.19956/) (2025)
- [SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents](https://mlsanity.com/paper/2605.14205/) (2026)
- [Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat: A Prominence-Stratified Cross-Provider Audit](https://mlsanity.com/paper/2605.30207/) (2026)
- [Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs](https://mlsanity.com/paper/2607.19243/) (2026)
- [Personalization, Personas, and Forecasting in Value Alignment](https://mlsanity.com/paper/2607.24782/) (2026)
- [Role Steering of Language Models for Social Simulations](https://mlsanity.com/paper/2608.00023/) (2026)
- [When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability](https://mlsanity.com/paper/2609.07987/) (2026)
- [Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses](https://mlsanity.com/paper/2511.23041/) (2025)
- [Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels](https://mlsanity.com/paper/2512.07474/) (2025)
- [How Value Induction Reshapes LLM Behaviour](https://mlsanity.com/paper/2605.07925/) (2026)
- [In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores](https://mlsanity.com/paper/2605.12530/) (2026)
- [Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents](https://mlsanity.com/paper/2605.15473/) (2026)
- [When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detection](https://mlsanity.com/paper/2605.27313/) (2026)
- [Occupational Prompting Reveals Cultural Bias in Large Language Models](https://mlsanity.com/paper/2606.12443/) (2026)

## Abstract (arXiv)

Persona prompting is increasingly used in large language models (LLMs) to simulate views of various sociodemographic groups. However, how a persona prompt is formulated can significantly affect outcomes, raising concerns about the fidelity of such simulations. Using five open-source LLMs, we systematically examine how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence LLM simulations across 15 intersectional demographic groups in both open- and closed-ended tasks. Our findings show that LLMs struggle to simulate marginalized groups but that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment. Surprisingly, smaller models like OLMo-2-7B outperform larger ones such as Llama-3.3-70B. Our findings offer actionable guidance for designing sociodemographic persona prompts in LLM-based simulation studies.