# Benchmarking large language model agent societies against human behavioural distributions

arXiv: https://arxiv.org/abs/2608.28182 · HTML: https://mlsanity.com/paper/2608.28182/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 30)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?](https://mlsanity.com/paper/2406.13605/) (2024)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing](https://mlsanity.com/paper/2309.05898/) (2023)
- [Emergence of Social Norms in Generative Agent Societies: Principles and Architecture](https://mlsanity.com/paper/2403.08251/) (2024)
- [TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs](https://mlsanity.com/paper/2410.10479/) (2024)
- [GLEE: A Unified Framework and Benchmark for Language-based Economic Environments](https://mlsanity.com/paper/2410.05254/) (2024)
- [The Spontaneous Emergence of Conventions: An Experimental Study of Cultural Evolution](https://mlsanity.com/paper/1502.06910/) (2015)
- [LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory](https://mlsanity.com/paper/2502.20432/) (2025)
- [The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies](https://mlsanity.com/paper/2509.18052/) (2025)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [Emergent LLM behaviors are observationally equivalent to data leakage](https://mlsanity.com/paper/2505.23796/) (2025)
- [M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Games](https://mlsanity.com/paper/2601.08462/) (2026)
- [Stop Drawing Scientific Claims from LLM Social Simulations Without Robustness Audits](https://mlsanity.com/paper/2605.18890/) (2026)
- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)
- [Large Language Models as Psychological Simulators: A Methodological Guide](https://mlsanity.com/paper/2506.16702/) (2025)
- [Reply to "Emergent LLM behaviors are observationally equivalent to data leakage"](https://mlsanity.com/paper/2506.18600/) (2025)

## Abstract (arXiv)

Populations of large language model agents are increasingly used as experimental societies. Three doubts shadow every such result: whether the agents behave like the humans they stand in for, whether a finding survives changes to the apparatus that leave the rules untouched, and whether apparent social dynamics are interaction at all rather than the reproduction of experiments the models have read. This article introduces SILICA, an open instrument that tests all three. Five environments carry published human anchors, each paired with perturbations that re-render the same rules and with variants whose payoffs point away from the memorised result. Twelve open-weight models were run through it on a single consumer graphics card. Agreement with human data is confined to starting points: first-round public-goods contributions fall inside the equivalence margin for eight of eleven models, while no model matches end-state contributions or the human corridor of cooperation. Merely swapping the order in which two actions are listed costs one model 58 points of cooperation. Presenting responders with a fixed schedule of offers shows that only one model, the sole reasoning-trained one, places its acceptance threshold where the incentive requires; two move theirs part of the way, two move them the wrong way, and three never acquire one. Conventions form through a shared prior over the names rather than through negotiation, though negotiation reappears once that prior is disrupted. On the certification ladder defined here, current silicon societies support exploratory claims and no more.