mlsanityData to 7 Oct 2026

Evaluating LLMs as Human Surrogates in Controlled Experiments

arXiv 2604.15329 ↗First posted 8 Mar 2026cs.HC cs.AI cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 14 of its 36 references that are in the map. First posted on arXiv 8 Mar 2026.

Builds on

14 of 36 references.

References · 14

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  7. 2022Using cognitive psychology to understand GPT-3cited by 389
  8. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  9. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  10. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  11. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  12. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 29
2 more
  1. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarioscited by 27
  2. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence?
  2. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina
  3. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
  4. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications
  5. 2025Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management
  6. 2023Do LLMs exhibit human-like response biases? A case study in survey design
  7. 2024Simulating Field Experiments with Large Language Models
  8. 2026LLMs Can Better Capture Human Judgments--With the Right Prompts