mlsanityData to 14 Oct 2026

Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification

arXiv 2510.11408 ↗First posted 13 Oct 2025cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Surveys provide valuable insights into public opinion and behavior, but their execution is costly and slow. Large language models (LLMs) have been proposed as a scalable, low-cost substitute for human respondents, but their outputs are often biased and yield invalid estimates. We study the interplay between synthesis methods that use LLMs to generate survey responses and rectification methods that debias population estimates, and explore how human responses are best allocated between them. Using two panel surveys with questions on nutrition, politics, and economics, we find that synthesis alone introduces substantial bias (24-86%), whereas combining it with rectification reduces bias below 5% and increases effective sample size by up to 14%. Overall, we challenge the common practice of using all human responses for fine-tuning, showing that under a fixed budget, allocating most to rectification results in far more effective estimation.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

42 of 73 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Whose Opinions Do Language Models Reflect?cited by 773
  5. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  6. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  7. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  8. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  9. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  10. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  11. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  12. 2024Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Modelscited by 162
30 more
  1. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  2. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  3. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  4. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  5. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  6. 2024The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Ratescited by 69
  7. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  8. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 58
  9. 2023On the steerability of large language models toward data-driven personascited by 53
  10. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 53
  11. 2023Multimodal LLMs for health grounded in individual-specific datacited by 49
  12. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 49
  13. 2023Grounding Gaps in Language Model Generationscited by 47
  14. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  15. 2025Creating General User Models from Computer Usecited by 44
  16. 2025Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforcecited by 38
  17. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  18. 2020Data Leverage: A Framework for Empowering the Public in its Relationship with Technology Companiescited by 26
  19. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 20
  20. 2025Mixture-of-Personas Language Models for Population Simulationcited by 20
  21. 2020Control, Generate, Augment: A Scalable Framework for Multi-Attribute Text Generationcited by 18
  22. 2024LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanationscited by 16
  23. 2022Spillover of Antisocial Behavior from Fringe Platforms: The Unintended Consequences of Community Banningcited by 14
  24. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 14
  25. 2025LLM Bidders Preserve the Mechanism-Level Orderings of Human Bidderscited by 10
  26. 2024Are Large Language Models Chameleons? An Attempt to Simulate Social Surveyscited by 8
  27. 2025No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inferencecited by 8
  28. 2025Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessmentcited by 4
  29. 2025Does Content Moderation Lead Users Away from Fringe Movements? Evidence from a Recovery Communitycited by 4
  30. 2025Meat-Free Day Reduces Greenhouse Gas Emissions but Poses Challenges for Customer Retention and Adherence to Dietary Guidelinescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 16.

18 papers.

Sort:

By area:

  1. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  2. 2026Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Datacited by 3
  3. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?cited by 2
  4. 2026Mitigating LLM-based p-Hacking by Preregistering for the Next LLMcited by 2
  5. 2025Graph-Based Alternatives to LLMs for Human Simulationcited by 1
  6. 2026Multi-Perspective LLM Annotations for Valid Analyses in Subjective Taskscited by 1
  7. 2026In-Context Learning for the Imputation of Public Opinion Data with Large Language Modelscited by 1
  8. 2026Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogatescited by 1
  9. 2026When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutabilitycited by 1
  10. 2026When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panelscited by 1
  11. 2026The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variancecited by 1
  12. 2026Consequential Behaviour and Representational Fairness in the Validation of Synthetic Researchcited by 1
6 more
  1. 2026PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMscited by 0
  2. 2026Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Modelcited by 0
  3. 2026Hybrid Panels: Toward Human-AI Collaboration in Survey Researchcited by 0
  4. 2026Augmented Hypothesis Testing with Persona-Based LLM Simulationscited by 0
  5. 2026Simulating Respondents, Not Single Questions: Coherent Survey Generation with Large Language Modelscited by 0
  6. 2026Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristicscited by 0