# Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management

arXiv: https://arxiv.org/abs/2510.03310 · HTML: https://mlsanity.com/paper/2510.03310/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning](https://mlsanity.com/paper/1908.08729/) (2019)
- [EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities](https://mlsanity.com/paper/2310.10436/) (2023)
- [Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs](https://mlsanity.com/paper/2407.01082/) (2024)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions](https://mlsanity.com/paper/2502.16761/) (2025)
- [Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://mlsanity.com/paper/2502.07068/) (2025)
- [Wireless Multi-Agent Generative AI: From Connected Intelligence to Collective Intelligence](https://mlsanity.com/paper/2307.02757/) (2023)
- [SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Data](https://mlsanity.com/paper/2309.13193/) (2023)
- [Large Language Models for Market Research: A Data-augmentation Approach](https://mlsanity.com/paper/2412.19363/) (2024)
- [Are Large Language Models (LLMs) Good Social Predictors?](https://mlsanity.com/paper/2402.12620/) (2024)

## Built on by (most cited first, 2 of 2)

- [OptiRepair: Closed-Loop Diagnosis and Repair of Supply Chain Optimization Models with LLM Agents](https://mlsanity.com/paper/2602.19439/) (2026)
- [UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs](https://mlsanity.com/paper/2606.06622/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to simulate human behavior in business, economics, and the social sciences, offering a low-cost complement to laboratory experiments, field studies, and surveys. This paper evaluates how well LLMs replicate human behavior in operations management. Using nine published behavioral-operations experiments, we assess LLM performance along two dimensions: whether LLM-generated data reproduce the original hypothesis-test outcomes, and whether their full response distributions align with human data, measured by Wasserstein distance. We find that LLMs often replicate hypothesis-level effects, suggesting that they can capture salient decision biases and behavioral regularities. However, their response distributions frequently diverge from human data, even for strong proprietary models, with dispersion mismatch playing an important role. We also examine two lightweight mitigation strategies: chain-of-thought prompting and hyperparameter tuning. Both can reduce distributional misalignment, and appropriate tuning can sometimes allow smaller or open-source models to match or outperform larger proprietary systems.