# Large Language Models as Virtual Survey Respondents: Evaluating Sociodemographic Response Generation

arXiv: https://arxiv.org/abs/2509.06337 · HTML: https://mlsanity.com/paper/2509.06337/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews](https://mlsanity.com/paper/2310.17976/) (2023)
- [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://mlsanity.com/paper/2502.08235/) (2025)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models](https://mlsanity.com/paper/2307.16180/) (2023)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information](https://mlsanity.com/paper/2402.18144/) (2024)
- [AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction](https://mlsanity.com/paper/2305.09620/) (2023)
- [How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO](https://mlsanity.com/paper/2404.13957/) (2024)
- [Few-Shot Learning with Class Imbalance](https://mlsanity.com/paper/2101.02523/) (2021)
- [Human Simulacra: Benchmarking the Personification of Large Language Models](https://mlsanity.com/paper/2402.18180/) (2024)
- [Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing](https://mlsanity.com/paper/2405.07726/) (2024)
- [Are Large Language Models (LLMs) Good Social Predictors?](https://mlsanity.com/paper/2402.12620/) (2024)
- [Large Language Models as Instruments of Power: New Regimes of Autonomous Manipulation and Control](https://mlsanity.com/paper/2405.03813/) (2024)
- [ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks](https://mlsanity.com/paper/2412.09173/) (2024)
- [On the Generalization vs Fidelity Paradox in Knowledge Distillation](https://mlsanity.com/paper/2505.15442/) (2025)
- [PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traits](https://mlsanity.com/paper/2401.07363/) (2024)

## Built on by (most cited first, 7 of 7)

- [From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes](https://mlsanity.com/paper/2605.16303/) (2026)
- [In-Context Learning for the Imputation of Public Opinion Data with Large Language Models](https://mlsanity.com/paper/2606.09351/) (2026)
- [Using Zero-Shot LLM-Generated Survey Data for Geographically Explicit Population Synthesis](https://mlsanity.com/paper/2605.27401/) (2026)
- [Generating Public Health Responses using Survey-Augmented Large Language Models](https://mlsanity.com/paper/2606.21820/) (2026)
- [From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration](https://mlsanity.com/paper/2609.13261/) (2026)
- [Large Language Models Substantially Compress Well-Being Inequality but Largely Preserve Its Socioeconomic Structure](https://mlsanity.com/paper/2609.33055/) (2026)
- [Simulating Respondents, Not Single Questions: Coherent Survey Generation with Large Language Models](https://mlsanity.com/paper/2609.34828/) (2026)

## Abstract (arXiv)

Questionnaire-based surveys are foundational to social science research and public policymaking, yet traditional survey methods remain costly, time-consuming, and often limited in scale. Although prior work has explored large language models (LLMs) as virtual survey respondents, existing studies often address narrow task settings, focus on single sociological domains, or lack a unified evaluation framework that enables systematic comparison across diverse datasets and models. To address these gaps, we introduce two complementary task abstractions: Partial Attribute Simulation (PAS), where LLMs predict missing attributes from incomplete respondent profiles, and Full Attribute Simulation (FAS), where LLMs generate complete synthetic datasets under zero-context and context-enhanced conditions. Both are framed as diagnostic and exploratory tools rather than replacements for human data collection. We curate LLM-S^3 (Large Language Model-based Sociodemographic Survey Simulation), a benchmark spanning 11 real-world public datasets across four sociological domains, and evaluate GPT-3.5/4 Turbo and LLaMA 3.0/3.1-8B under zero-shot and few-shot settings. Our evaluation reveals consistent performance trends across model families, highlights failure modes in structured output generation, and demonstrates how context and prompt design affect simulation fidelity. Our code and dataset are available at: https://github.com/dart-lab-research/LLM-S-Cube-Benchmark