# The threat of analytic flexibility in using large language models to simulate human data

arXiv: https://arxiv.org/abs/2509.13397 · HTML: https://mlsanity.com/paper/2509.13397/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 68 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions](https://mlsanity.com/paper/2502.16761/) (2025)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information](https://mlsanity.com/paper/2402.18144/) (2024)
- [Large language models predict human sensory judgments across six modalities](https://mlsanity.com/paper/2302.01308/) (2023)
- [Diminished Diversity-of-Thought in a Standard Large Language Model](https://mlsanity.com/paper/2302.07267/) (2023)
- [The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models](https://mlsanity.com/paper/2507.16076/) (2025)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Six Fallacies in Substituting Large Language Models for Human Participants](https://mlsanity.com/paper/2402.04470/) (2024)
- [LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses](https://mlsanity.com/paper/2406.04755/) (2024)
- [Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?](https://mlsanity.com/paper/2506.09796/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [ChatGPT is not A Man but Das Man: Representativeness and Structural Consistency of Silicon Samples Generated by Large Language Models](https://mlsanity.com/paper/2507.02919/) (2025)
- [Simulating Tabular Datasets through LLMs to Rapidly Explore Hypotheses about Real-World Entities](https://mlsanity.com/paper/2411.18071/) (2024)
- [Identifying Non-Replicable Social Science Studies with Language Models](https://mlsanity.com/paper/2503.10671/) (2025)

## Built on by (most cited first, 15 of 15)

- [Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models](https://mlsanity.com/paper/2510.11586/) (2025)
- [Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs](https://mlsanity.com/paper/2510.14242/) (2025)
- [Addressing Longstanding Challenges in Cognitive Science with Language Models](https://mlsanity.com/paper/2511.00206/) (2025)
- [Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Data](https://mlsanity.com/paper/2606.28963/) (2026)
- [Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation](https://mlsanity.com/paper/2510.13884/) (2025)
- [QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models](https://mlsanity.com/paper/2512.08646/) (2025)
- [RegCheck: A tool for structured comparisons between study registrations and papers](https://mlsanity.com/paper/2601.13330/) (2026)
- [The conditional superiority of fast silicon sampling](https://mlsanity.com/paper/2608.14079/) (2026)
- [Assessing mentalization in humans and large language models](https://mlsanity.com/paper/2608.26291/) (2026)
- [When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability](https://mlsanity.com/paper/2609.07987/) (2026)
- [Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Bandits](https://mlsanity.com/paper/2604.02527/) (2026)
- [Validating LLMs in social science: Epistemic threats and emerging norms](https://mlsanity.com/paper/2607.07915/) (2026)
- [Human-like moral judgments conceal divergent motive attributions in large language models](https://mlsanity.com/paper/2609.07353/) (2026)
- [Benchmarking Psychological Dynamics in Generative Agents](https://mlsanity.com/paper/2610.04246/) (2026)
- [Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents](https://mlsanity.com/paper/2610.14912/) (2026)

## Abstract (arXiv)

Social scientists are now using large language models to create "silicon samples": synthetic datasets intended to stand in for human respondents. However, producing these samples requires many analytic choices, including model selection, sampling parameters, prompt format, and the amount of demographic or contextual information provided. Across two studies, I examine whether these choices materially affect correspondence between silicon samples and human data. In Study 1, I generated 252 silicon-sample configurations for a controlled case study using two social-psychological scales, evaluating whether configurations recovered participant rankings, response distributions, and between-scale correlations. Configurations varied substantially across all three criteria, and configurations that performed well on one dimension often performed poorly on another. In Study 2, I extended this analysis to a published silicon-sample use case by re-examining Argyle et al.'s (2023) Study 3 using 66 alternative configurations. Correlations between human and silicon association structures differed substantially across configurations, from r = .23 to r = .84. Taken together, the results from these studies demonstrate that different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples. I call for greater attention to the threat of analytic flexibility in using silicon samples and outline strategies that researchers may adopt to reduce this threat.