# The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective

arXiv: https://arxiv.org/abs/2312.15524 · HTML: https://mlsanity.com/paper/2312.15524/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 24 Dec 2023.

## Builds on (most cited first, 8 of 8)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings](https://mlsanity.com/paper/2003.11515/) (2020)
- [Large Language Models as Subpopulation Representative Models: A Review](https://mlsanity.com/paper/2310.17888/) (2023)
- [The Econometrics of Randomized Experiments](https://mlsanity.com/paper/1607.00698/) (2016)

## Built on by (most cited first, 30 of 59)

- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [LLM-based Human Simulations Have Not Yet Been Reliable](https://mlsanity.com/paper/2501.08579/) (2025)
- [Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions](https://mlsanity.com/paper/2505.17479/) (2025)
- [Large Language Models for Market Research: A Data-augmentation Approach](https://mlsanity.com/paper/2412.19363/) (2024)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Can LLMs Reliably Simulate Human Learner Actions? A Simulation Authoring Framework for Open-Ended Learning Environments](https://mlsanity.com/paper/2410.02110/) (2024)
- [Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue](https://mlsanity.com/paper/2409.08330/) (2024)
- [Perceptions of Linguistic Uncertainty by Language Models and Humans](https://mlsanity.com/paper/2407.15814/) (2024)
- [The Need for a Socially-Grounded Persona Framework for User Simulation](https://mlsanity.com/paper/2601.07110/) (2026)
- [RealTCD: Temporal Causal Discovery from Interventional Data with Large Language Model](https://mlsanity.com/paper/2404.14786/) (2024)
- [Explaining Large Language Models Decisions Using Shapley Values](https://mlsanity.com/paper/2404.01332/) (2024)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [IMPersona: Evaluating Individual Level LM Impersonation](https://mlsanity.com/paper/2504.04332/) (2025)
- [A validity-guided workflow for robust large language model research in psychology](https://mlsanity.com/paper/2507.04491/) (2025)
- [Using LLMs to Model the Beliefs and Preferences of Targeted Populations](https://mlsanity.com/paper/2403.20252/) (2024)
- [Prediction-Guided Active Experiments](https://mlsanity.com/paper/2411.12036/) (2024)
- [Aligning Large Language Model Agents with Rational and Moral Preferences: A Supervised Fine-Tuning Approach](https://mlsanity.com/paper/2507.20796/) (2025)
- [Overstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibility](https://mlsanity.com/paper/2602.04674/) (2026)
- [The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study](https://mlsanity.com/paper/2605.20767/) (2026)
- [Can Deception Detection Go Deeper? Dataset, Evaluation, and Benchmark for Deception Reasoning](https://mlsanity.com/paper/2402.11432/) (2024)
- [Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions](https://mlsanity.com/paper/2502.13135/) (2025)
- [Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experiments](https://mlsanity.com/paper/2505.09901/) (2025)
- [Reasoning and the Trusting Behavior of DeepSeek and GPT: An Experiment Revealing Hidden Fault Lines in Large Language Models](https://mlsanity.com/paper/2502.12825/) (2025)
- [Can LLMs Help Improve Analogical Reasoning For Strategic Decisions? Experimental Evidence from Humans and GPT-4](https://mlsanity.com/paper/2505.00603/) (2025)
- [From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images](https://mlsanity.com/paper/2511.22805/) (2025)
- [Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence](https://mlsanity.com/paper/2604.14575/) (2026)
- [From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes](https://mlsanity.com/paper/2605.16303/) (2026)
- [Deploying AI for Signal Processing education: Selected challenges and intriguing opportunities](https://mlsanity.com/paper/2509.08950/) (2025)
- [Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)](https://mlsanity.com/paper/2510.02331/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have shown impressive potential to simulate human behavior. We identify a fundamental challenge in using them to simulate experiments: when LLM-simulated subjects are blind to the experimental design (as is standard practice with human subjects), variations in treatment systematically affect unspecified variables that should remain constant, violating the unconfoundedness assumption. Using demand estimation as a context and an actual experiment with 40 different products as a benchmark, we show this can lead to implausible results. While confounding may in principle be addressed by controlling for covariates, this can compromise ecological validity in the context of LLM simulations: controlled covariates become artificially salient in the simulated decision process. We show formally that confoundness stems from ambiguous prompting strategies. Therefore, it can be addressed by developing unambiguous prompting strategies through unblinding, i.e., revealing the experiment design in LLM simulations. Our empirical results show that this strategy consistently enhances model performance across all tested models, including both out-of-box reasoning and non-reasoning models. We also show that it is a technique that complements fine-tuning: while fine-tuning can improve simulation performance, an unambiguous prompting strategy makes the predictions robust to the inclusion of irrelevant data in the fine-tuning process.