# Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents

arXiv: https://arxiv.org/abs/2602.18462 · HTML: https://mlsanity.com/paper/2602.18462/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 28 of 28)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generation](https://mlsanity.com/paper/2402.14744/) (2024)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction](https://mlsanity.com/paper/2305.09620/) (2023)
- [Y Social: an LLM-powered Social Media Digital Twin](https://mlsanity.com/paper/2408.00818/) (2024)
- [Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs](https://mlsanity.com/paper/2311.09730/) (2023)
- [The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks](https://mlsanity.com/paper/2304.13861/) (2023)
- [Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets](https://mlsanity.com/paper/2410.07991/) (2024)
- [Population-Aligned Persona Generation for LLM-based Social Simulation](https://mlsanity.com/paper/2509.10127/) (2025)
- [Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models](https://mlsanity.com/paper/2401.07115/) (2024)
- [Contextualized Counterspeech: Strategies for Adaptation, Personalization, and Evaluation](https://mlsanity.com/paper/2412.07338/) (2024)
- [Psychological Steering in LLMs: An Evaluation of Effectiveness and Trustworthiness](https://mlsanity.com/paper/2510.04484/) (2025)
- [Simulation of Language Evolution under Regulated Social Media Platforms: A Synergistic Approach of Large Language Models and Genetic Algorithms](https://mlsanity.com/paper/2502.19193/) (2025)
- [SimSpark: Interactive Simulation of Social Media Behaviors](https://mlsanity.com/paper/2506.14476/) (2025)
- [Evaluation of Reliability Criteria for News Publishers with Large Language Models](https://mlsanity.com/paper/2412.15896/) (2024)
- [When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generation](https://mlsanity.com/paper/2505.24613/) (2025)
- [Do Persona-Infused LLMs Affect Performance in a Strategic Reasoning Game?](https://mlsanity.com/paper/2512.06867/) (2025)
- [AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys](https://mlsanity.com/paper/2511.07871/) (2025)

## Built on by (most cited first, 10 of 10)

- [When Can Digital Personas Reliably Approximate Human Survey Findings?](https://mlsanity.com/paper/2605.10659/) (2026)
- [Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling](https://mlsanity.com/paper/2607.18310/) (2026)
- [When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels](https://mlsanity.com/paper/2609.13148/) (2026)
- [Should LLM Agents Decide in Social Simulations? Comparing Finite-State and LLM-Based Decision Policies](https://mlsanity.com/paper/2606.12369/) (2026)
- [When Persona Simulations Are Informative: Graph-Structured Signals for Pluralistic Opinion Sensing](https://mlsanity.com/paper/2608.22438/) (2026)
- [Distributional Validity of a Korean Synthetic Persona Panel: Evidence From the Korea Media Panel Survey](https://mlsanity.com/paper/2608.28615/) (2026)
- [From Simulated Citizens to Simulated Deliberation: Challenges in Representation and Interaction](https://mlsanity.com/paper/2609.07573/) (2026)
- [Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populations](https://mlsanity.com/paper/2609.29928/) (2026)
- [Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas](https://mlsanity.com/paper/2610.03998/) (2026)
- [When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark](https://mlsanity.com/paper/2610.16067/) (2026)

## Abstract (arXiv)

Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses.