# Towards Operational Validation of LLM-Agent Social Simulations: A Replicated Study of a Reddit-like Technology Forum

arXiv: https://arxiv.org/abs/2508.21740 · HTML: https://mlsanity.com/paper/2508.21740/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 65 references that are in the map.
- First posted on arXiv 29 Aug 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 26 of 26)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [Sycophancy in Large Language Models: Causes and Mitigations](https://mlsanity.com/paper/2411.15287/) (2024)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [Machine Culture](https://mlsanity.com/paper/2311.11388/) (2023)
- [Y Social: an LLM-powered Social Media Digital Twin](https://mlsanity.com/paper/2408.00818/) (2024)
- [A new sociology of humans and machines](https://mlsanity.com/paper/2402.14410/) (2024)
- [Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations](https://mlsanity.com/paper/2504.03274/) (2025)
- [Six Fallacies in Substituting Large Language Models for Human Participants](https://mlsanity.com/paper/2402.04470/) (2024)
- [Origins of power-law degree distribution in the heterogeneity of human activity in social networks](https://mlsanity.com/paper/1304.4523/) (2013)
- [A Clarified Typology of Core-Periphery Structure in Networks](https://mlsanity.com/paper/2005.10191/) (2020)
- [Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity](https://mlsanity.com/paper/2309.06364/) (2023)
- [Are LLM-Powered Social Media Bots Realistic?](https://mlsanity.com/paper/2508.00998/) (2025)
- [Can We Fix Social Media? Testing Prosocial Interventions using Generative Social Simulation](https://mlsanity.com/paper/2508.03385/) (2025)
- ["I Can't Keep It Up." A Dataset from the Defunct Voat.co News Aggregator](https://mlsanity.com/paper/2201.05933/) (2022)
- [Multi-Platform Aggregated Dataset of Online Communities (MADOC)](https://mlsanity.com/paper/2501.12886/) (2025)
- [Post-Post-API Age: Studying Digital Platforms in Scant Data Access Times](https://mlsanity.com/paper/2505.09877/) (2025)

## Built on by (most cited first, 2 of 2)

- [Agentic Microphysics: A Manifesto for Generative AI Safety](https://mlsanity.com/paper/2604.15236/) (2026)
- [When Is Emergent Consensus Real? A Measured Coupling Gain and a Validity Diagnostic for LLM Agent Societies](https://mlsanity.com/paper/2606.22203/) (2026)

## Abstract (arXiv)

Validation of LLM-agent social simulations remains underdeveloped, with most studies relying on subjective assessments or single runs. We address this gap by running 30 independent 30-day simulations of a technology forum modeled on Voat's v/technology, using stateless Dolphin Mistral 24B agents on the Y Social platform, and evaluating operational validity across five dimensions: activity patterns, network structure, toxicity, topical coverage, and stylistic convergence. Against 30 matched, non-overlapping 30-day Voat comparison windows, results show overlapping 99% confidence intervals for unique users, root posts, and daily active users, while comments, average thread length, and mean toxicity remain higher in simulation. Both simulated and empirical networks exhibit core-periphery structure, though simulated cores are larger and more diffuse and repeated interactions are less frequent. Topic alignment is near-complete, but toxicity is misallocated across content layers: simulated root posts are substantially more toxic than real submissions, while simulated comments are less toxic than Voat comments. These findings demonstrate that LLM agents in platform-faithful environments can reproduce familiar online regularities, while systematic divergences, particularly those linked to stateless agent design and content-layer calibration, point to concrete directions for future improvement.