# Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations

arXiv: https://arxiv.org/abs/2504.03274 · HTML: https://mlsanity.com/paper/2504.03274/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 59 of its 118 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 59)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models](https://mlsanity.com/paper/2304.03738/) (2023)
- [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023)
- [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023)
- [Theory of Mind for Multi-Agent Collaboration via Large Language Models](https://mlsanity.com/paper/2310.10701/) (2023)
- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [OASIS: Open Agent Social Interaction Simulations with One Million Agents](https://mlsanity.com/paper/2411.11581/) (2024)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming](https://mlsanity.com/paper/2310.06500/) (2023)

## Built on by (most cited first, 28 of 28)

- [The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies](https://mlsanity.com/paper/2509.18052/) (2025)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [Can Generative AI agents behave like humans? Evidence from laboratory market experiments](https://mlsanity.com/paper/2505.07457/) (2025)
- [Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale](https://mlsanity.com/paper/2505.23785/) (2025)
- [Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research](https://mlsanity.com/paper/2506.01839/) (2025)
- [Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism](https://mlsanity.com/paper/2506.21974/) (2025)
- [textttBluePrint: A Social Media User Dataset for LLM Persona Evaluation and Training](https://mlsanity.com/paper/2510.02343/) (2025)
- [Computational Turing Test Reveals Systematic Differences Between Human and AI Language](https://mlsanity.com/paper/2511.04195/) (2025)
- [CoBAD: Modeling Collective Behaviors for Human Mobility Anomaly Detection](https://mlsanity.com/paper/2508.14088/) (2025)
- [Large Language Models for Agent-Based Modelling: Current and possible uses across the modelling cycle](https://mlsanity.com/paper/2507.05723/) (2025)
- [The Influence of Human-inspired Agentic Sophistication in LLM-driven Strategic Reasoners](https://mlsanity.com/paper/2505.09396/) (2025)
- [Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives](https://mlsanity.com/paper/2506.12242/) (2025)
- [Emergent Crowds Dynamics from Language-Driven Multi-Agent Interactions](https://mlsanity.com/paper/2508.15047/) (2025)
- [Towards Operational Validation of LLM-Agent Social Simulations: A Replicated Study of a Reddit-like Technology Forum](https://mlsanity.com/paper/2508.21740/) (2025)
- [Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulations](https://mlsanity.com/paper/2510.13982/) (2025)
- [Large language models replicate and predict human cooperation across experiments in game theory](https://mlsanity.com/paper/2511.04500/) (2025)
- [The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions](https://mlsanity.com/paper/2511.08592/) (2025)
- [When Plausible Is Not Realistic: Evaluating Human Mobility in LLM-Based Urban Simulation](https://mlsanity.com/paper/2606.13835/) (2026)
- [Integrating Machine Learning into Belief-Desire-Intention Agents: Current Advances and Open Challenges](https://mlsanity.com/paper/2510.20641/) (2025)
- [Validating Generative Agent-Based Models for Logistics and Supply Chain Management Research](https://mlsanity.com/paper/2508.20234/) (2025)
- [Next Reply Prediction X Dataset: Linguistic Discrepancies in Naively Generated Content](https://mlsanity.com/paper/2602.19177/) (2026)
- [The textitSilicon Society Cookbook: Design Space of LLM-based Social Simulations](https://mlsanity.com/paper/2605.00197/) (2026)
- [Mechanism Plausibility in Generative Agent-Based Modeling](https://mlsanity.com/paper/2605.12824/) (2026)
- [The Contagion Tensor: A Framework for Measuring Output-Distribution Coupling in Multi-Agent LLM Systems -- and Auditing the Claims It Enables](https://mlsanity.com/paper/2606.28839/) (2026)
- [A Large Language Model-Driven Agent-Based Modeling Framework with Multi-Round Communication for Simulating Vaccine Opinion Dynamics](https://mlsanity.com/paper/2607.07387/) (2026)
- [Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions](https://mlsanity.com/paper/2607.17437/) (2026)
- [Modeling Social Dynamics with an LLM-Enabled Agent Based Network-Dynamic (LAND) Model](https://mlsanity.com/paper/2608.00929/) (2026)
- [Illusory Truth or Mere Exposure? Model-Dependent Repetition Effects in LLM-Based Social Media Simulations](https://mlsanity.com/paper/2609.36278/) (2026)

## Abstract (arXiv)

Recent advancements in AI have reinvigorated Agent-Based Models (ABMs), as the integration of Large Language Models (LLMs) has led to the emergence of ``generative ABMs'' as a novel approach to simulating social systems. While ABMs offer means to bridge micro-level interactions with macro-level patterns, they have long faced criticisms from social scientists, pointing to e.g., lack of realism, computational complexity, and challenges of calibrating and validating against empirical data. This paper reviews the generative ABM literature to assess how this new approach adequately addresses these long-standing criticisms. Our findings show that studies show limited awareness of historical debates. Validation remains poorly addressed, with many studies relying solely on subjective assessments of model `believability', and even the most rigorous validation failing to adequately evidence operational validity. We argue that there are reasons to believe that LLMs will exacerbate rather than resolve the long-standing challenges of ABMs. The black-box nature of LLMs moreover limit their usefulness for disentangling complex emergent causal mechanisms. While generative ABMs are still in a stage of early experimentation, these findings question of whether and how the field can transition to the type of rigorous modeling needed to contribute to social scientific theory.