mlsanityData to 7 Oct 2026

Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents

arXiv 2602.18462 ↗First posted 6 Feb 2026cs.CY cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

28 of 41 references.

References · 28

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  4. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  5. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  6. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  7. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  8. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  9. 2024Investigating Cultural Alignment of Large Language Modelscited by 140
  10. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  11. 2024Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generationcited by 99
  12. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
16 more
  1. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 50
  2. 2023AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Predictioncited by 46
  3. 2024Y Social: an LLM-powered Social Media Digital Twincited by 43
  4. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  5. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  6. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  7. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  8. 2024Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Modelscited by 19
  9. 2024Contextualized Counterspeech: Strategies for Adaptation, Personalization, and Evaluationcited by 15
  10. 2025Psychological Steering in LLMs: An Evaluation of Effectiveness and Trustworthinesscited by 7
  11. 2025Simulation of Language Evolution under Regulated Social Media Platforms: A Synergistic Approach of Large Language Models and Genetic Algorithmscited by 5
  12. 2025SimSpark: Interactive Simulation of Social Media Behaviorscited by 5
  13. 2024Evaluation of Reliability Criteria for News Publishers with Large Language Modelscited by 4
  14. 2025When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generationcited by 4
  15. 2025Do Persona-Infused LLMs Affect Performance in a Strategic Reasoning Game?cited by 3
  16. 2025AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveyscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026When Can Digital Personas Reliably Approximate Human Survey Findings?cited by 2
  2. 2026Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modelingcited by 1
  3. 2026Should LLM Agents Decide in Social Simulations? Comparing Finite-State and LLM-Based Decision Policiescited by 0
  4. 2026When Persona Simulations Are Informative: Graph-Structured Signals for Pluralistic Opinion Sensingcited by 0
  5. 2026Distributional Validity of a Korean Synthetic Persona Panel: Evidence From the Korea Media Panel Surveycited by 0
  6. 2026From Simulated Citizens to Simulated Deliberation: Challenges in Representation and Interactioncited by 0
  7. 2026Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populationscited by 0