mlsanityData to 14 Oct 2026

LLM-Based Social Simulations Require a Boundary

arXiv 2506.19806 ↗First posted 24 Jun 2025cs.CY cs.CL cs.MAcited by 12

Not on the map. Why?

Abstract (arXiv)

This position paper argues that LLM-based social simulations require clear boundaries to make meaningful contributions to social science. While Large Language Models (LLMs) offer promising capabilities for simulating human behavior, their tendency to produce homogeneous outputs, acting as an "average persona", fundamentally limits their ability to capture the behavioral diversity essential for complex social dynamics. We examine why heterogeneity matters for social simulations and how current LLMs fall short, analyzing the relationship between mean alignment and variance in LLM-generated behaviors. Through a systematic review of representative studies, we find that validation practices often fail to match the heterogeneity requirements of research questions: while most papers include ground truth comparisons, fewer than half explicitly assess behavioral variance, and most that do report lower variance than human populations. We propose that researchers should: (1) match validation depth to the heterogeneity demands of their research questions, (2) explicitly report variance alongside mean alignment, and (3) constrain claims to collective-level qualitative patterns when variance is insufficient. Rather than dismissing LLM-based simulation, we advocate for a boundary-aware approach that ensures these methods contribute genuine insights to social science.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 100 of its 192 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

100 of 192 references.

References · 100

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  4. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  5. 2023Whose Opinions Do Language Models Reflect?cited by 773
  6. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  7. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  8. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  9. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  10. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  11. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  12. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
38 more
  1. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  2. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  3. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 249
  4. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 233
  5. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  6. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  7. 2023Playing repeated games with Large Language Modelscited by 217
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  9. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  10. 2023Personality Traits in Large Language Modelscited by 169
  11. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activitiescited by 154
  12. 2023Simulating Opinion Dynamics with Networks of LLM-based Agentscited by 154
  13. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 145
  14. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 144
  15. 2023User Behavior Simulation with Large Language Model based Agentscited by 140
  16. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  17. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 132
  18. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 127
  19. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 125
  20. 2024Emergent social conventions and collective bias in LLM populationscited by 123
  21. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  22. 2024Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generationcited by 100
  23. 2024Systematic Biases in LLM Simulations of Debatescited by 99
  24. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  25. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 97
  26. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  27. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  28. 2024Evaluating Large Language Model Biases in Persona-Steered Generationcited by 78
  29. 2023Strategic Reasoning with Language Modelscited by 64
  30. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 63
  31. 2023The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspectivecited by 59
  32. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  33. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 59
  34. 2025SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Userscited by 55
  35. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 53
  36. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 53
  37. 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 51
  38. 2024From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake Newscited by 51

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 21
  2. 2026Exploring Silicon-Based Societies: An Early Study of the Moltbook Agent Communitycited by 13
  3. 2025From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulationcited by 10
  4. 2026Overstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibilitycited by 5
  5. 2026Restoring Heterogeneity in LLM-based Social Simulation: An Audience Segmentation Approachcited by 4
  6. 2025The Emergence of Altruism in Large-Language-Model Agents Societycited by 2
  7. 2026Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agentscited by 1
  8. 2026AgoraSim: A Hybrid Agent-Based Modeling Frameworkcited by 1
  9. 2026Will Scaling Improve Social Simulation with LLMs?cited by 0
  10. 2026Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptionscited by 0
  11. 2026Benchmarking large language model agent societies against human behavioural distributionscited by 0
  12. 2026From Caricature to Collective Artifact: How Persona Construction Drives Polarization in Generative Social Simulationscited by 0