mlsanityData to 7 Oct 2026

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

arXiv 2208.10264 ↗First posted 18 Aug 2022cs.CL cs.AI cs.LGcited by 506

Not on the map. Why?

Cited by 134 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new type of test, called a Turing Experiment (TE), for evaluating to what extent a given language model, such as GPT models, can simulate different aspects of human behavior. A TE can also reveal consistent distortions in a language model's simulation of a specific human behavior. Unlike the Turing Test, which involves simulating a single arbitrary individual, a TE requires simulating a representative sample of participants in human subject research. We carry out TEs that attempt to replicate well-established findings from prior studies. We design a methodology for simulating TEs and illustrate its use to compare how well different language models are able to reproduce classic economic, psycholinguistic, and social psychology experiments: Ultimatum Game, Garden Path Sentences, Milgram Shock Experiment, and Wisdom of Crowds. In the first three TEs, the existing findings were replicated using recent models, while the last TE reveals a "hyper-accuracy distortion" present in some language models (including ChatGPT and GPT-4), which could affect downstream applications in education and the arts.

Key facts. Cited by 506 papers in the map (data through 7 Oct 2026). Builds on 19 of its 65 references that are in the map. First posted on arXiv 18 Aug 2022.

Builds on

19 of 65 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
7 more
  1. 2022Using cognitive psychology to understand GPT-3cited by 389
  2. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  3. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  4. 2022Language models show human-like content effects on reasoning taskscited by 171
  5. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  6. 2013Social Influence and the Collective Dynamics of Opinion Formationcited by 16
  7. 2017White, Man, and Highly Followed: Gender and Race Inequalities in Twittercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 54; 2024: 132; 2025: 155; 2026: 161.

506 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  5. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  6. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  7. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  8. 2023Challenges and Applications of Large Language Modelscited by 251
  9. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 229
  10. 2023Playing repeated games with Large Language Modelscited by 213
  11. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  12. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
38 more
  1. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 180
  2. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  3. 2024A Roadmap to Pluralistic Alignmentcited by 175
  4. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  5. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  6. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  7. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 125
  8. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  9. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  10. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  11. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  12. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  13. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 93
  14. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  15. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  16. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  17. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  18. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  19. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  20. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  21. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 71
  22. 2023Questioning the Survey Responses of Large Language Modelscited by 70
  23. 2023Turning large language models into cognitive modelscited by 66
  24. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  25. 2023Strategic Reasoning with Language Modelscited by 64
  26. 2023Building Emotional Support Chatbots in the Era of LLMscited by 62
  27. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  28. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 57
  29. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  30. 2023Do LLM Agents Exhibit Social Behavior?cited by 53
  31. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 50
  32. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  33. 2024Virtual Personas for Language Models via an Anthology of Backstoriescited by 49
  34. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 48
  35. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  36. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agentscited by 48
  37. 2024Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Informationcited by 48
  38. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48

50 most cited of 506 All 506 (JSON)