# Artificial Societies Benchmark: A Validation Framework for Synthetic Research

arXiv: https://arxiv.org/abs/2609.30030 · HTML: https://mlsanity.com/paper/2609.30030/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 40 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 9 of 9)

- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [Are Large Language Models Consistent over Value-laden Questions?](https://mlsanity.com/paper/2407.02996/) (2024)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions](https://mlsanity.com/paper/2505.17479/) (2025)
- [Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents](https://mlsanity.com/paper/2608.14606/) (2026)
- [SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models](https://mlsanity.com/paper/2510.11131/) (2025)
- [When Can Digital Personas Reliably Approximate Human Survey Findings?](https://mlsanity.com/paper/2605.10659/) (2026)
- [Consequential Behaviour and Representational Fairness in the Validation of Synthetic Research](https://mlsanity.com/paper/2609.27690/) (2026)

## Abstract (arXiv)

A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respond to changes in conditions. We introduce the Artificial Societies Benchmark to help researchers assess whether synthetic populations support their intended analyses. The framework combines eleven tests across internal, construct, and external validity, drawing on twenty human sources and comparing nine language models. It connects each research use to the evidence it requires and tests how results change with the information we supply about respondents. Importantly, strong performance in one domain does not establish fidelity in the others. Models often answer too consistently, compress response scales, and alter relationships between traits whilst richer profiles improve prediction for some models and worsen it for others. The resulting scorecard helps researchers identify which aspects of a synthetic population can support their analysis and where researchers need further human evidence.