# This human study did not involve human subjects: Validating LLM simulations as behavioral evidence

arXiv: https://arxiv.org/abs/2602.15785 · HTML: https://mlsanity.com/paper/2602.15785/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 115 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Builds on (most cited first, 30 of 55)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2402.13950/) (2024)
- [When combinations of humans and AI are useful: A systematic review and meta-analysis](https://mlsanity.com/paper/2405.06087/) (2024)
- [Are self-explanations from Large Language Models faithful?](https://mlsanity.com/paper/2401.07927/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Can Large Language Models Reason and Plan?](https://mlsanity.com/paper/2403.04121/) (2024)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- [Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions](https://mlsanity.com/paper/2502.16761/) (2025)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity](https://mlsanity.com/paper/2411.04427/) (2024)
- [LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language](https://mlsanity.com/paper/2405.12856/) (2024)

## Built on by (most cited first, 29 of 29)

- [Post-training makes large language models less human-like](https://mlsanity.com/paper/2605.07632/) (2026)
- [When Can Digital Personas Reliably Approximate Human Survey Findings?](https://mlsanity.com/paper/2605.10659/) (2026)
- [SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation](https://mlsanity.com/paper/2604.07513/) (2026)
- [Simulated Customers Never Walk Away: Decision Fidelity of LLM User Simulators Measured Against Real Purchase Outcomes](https://mlsanity.com/paper/2606.20708/) (2026)
- [Mitigating LLM-based p-Hacking by Preregistering for the Next LLM](https://mlsanity.com/paper/2606.27687/) (2026)
- [Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks](https://mlsanity.com/paper/2603.21404/) (2026)
- [Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments](https://mlsanity.com/paper/2604.02458/) (2026)
- [Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates](https://mlsanity.com/paper/2608.29455/) (2026)
- [When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability](https://mlsanity.com/paper/2609.07987/) (2026)
- [Consequential Behaviour and Representational Fairness in the Validation of Synthetic Research](https://mlsanity.com/paper/2609.27690/) (2026)
- [Social Meaning in Large Language Models: Structure, Magnitude, and Pragmatic Prompting](https://mlsanity.com/paper/2604.02512/) (2026)
- [Evaluating LLMs as Human Surrogates in Controlled Experiments](https://mlsanity.com/paper/2604.15329/) (2026)
- [Adaptive Querying with AI Persona Priors](https://mlsanity.com/paper/2605.00696/) (2026)
- [An Algebraic Exposition of the Theory of Dyadic Morality](https://mlsanity.com/paper/2605.16153/) (2026)
- [Personality engineering with AI agents: A new methodology for negotiation research](https://mlsanity.com/paper/2605.20554/) (2026)
- [Validating LLMs in social science: Epistemic threats and emerging norms](https://mlsanity.com/paper/2607.07915/) (2026)
- [Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach](https://mlsanity.com/paper/2607.26317/) (2026)
- [Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation](https://mlsanity.com/paper/2608.03044/) (2026)
- [AI Evaluation Should Work With Humans](https://mlsanity.com/paper/2608.13577/) (2026)
- [Modeling Human Behavior with Type Vectors Using AI](https://mlsanity.com/paper/2608.18265/) (2026)
- [An Agentic Approach for Active Data Collection, Travel Behavior Modeling, and Weather-Sensitive Demand Prediction](https://mlsanity.com/paper/2608.20320/) (2026)
- [Algorithm Validation as a Policy Audit: Evidence from Race-blind Charging](https://mlsanity.com/paper/2609.13174/) (2026)
- [Interpreting and Steering LLM Agents for Social Simulations](https://mlsanity.com/paper/2609.16436/) (2026)
- [Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making](https://mlsanity.com/paper/2609.22225/) (2026)
- [Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populations](https://mlsanity.com/paper/2609.29928/) (2026)
- [Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation](https://mlsanity.com/paper/2609.35036/) (2026)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)
- [Benchmarking Psychological Dynamics in Generative Agents](https://mlsanity.com/paper/2610.04246/) (2026)
- [Cluster-Robust Prediction-Powered Inference](https://mlsanity.com/paper/2610.09601/) (2026)

## Abstract (arXiv)

A growing literature uses large language models (LLMs) as synthetic participants to generate cost-effective and nearly instantaneous responses in social science experiments. However, there is limited guidance on when such simulations support valid inference about human behavior. We contrast two strategies for obtaining valid estimates of causal effects and clarify the assumptions under which each is suitable for exploratory versus confirmatory research. Heuristic approaches seek to establish that simulated and observed human behavior are interchangeable through prompt engineering, model fine-tuning, and other repair strategies designed to reduce LLM-induced inaccuracies. While useful for many exploratory tasks, heuristic approaches lack the formal statistical guarantees typically required for confirmatory research. In contrast, statistical calibration combines auxiliary human data with statistical adjustments to account for discrepancies between observed and simulated responses. Under explicit assumptions, statistical calibration preserves validity and provides more precise estimates of causal effects at lower cost than experiments that rely solely on human participants. Yet the potential of both approaches depends on how well LLMs approximate the relevant populations. We consider what opportunities are overlooked when researchers focus myopically on substituting LLMs for human participants in a study.