# Emergent LLM behaviors are observationally equivalent to data leakage

arXiv: https://arxiv.org/abs/2505.23796 · HTML: https://mlsanity.com/paper/2505.23796/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 5 of 5)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://mlsanity.com/paper/2312.12343/) (2023)

## Built on by (most cited first, 9 of 9)

- [Conformity and Social Impact on AI Agents](https://mlsanity.com/paper/2601.05384/) (2026)
- [The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies](https://mlsanity.com/paper/2509.18052/) (2025)
- [Addressing Longstanding Challenges in Cognitive Science with Language Models](https://mlsanity.com/paper/2511.00206/) (2025)
- [Reply to "Emergent LLM behaviors are observationally equivalent to data leakage"](https://mlsanity.com/paper/2506.18600/) (2025)
- [Large language models replicate and predict human cooperation across experiments in game theory](https://mlsanity.com/paper/2511.04500/) (2025)
- [When Is Emergent Consensus Real? A Measured Coupling Gain and a Validity Diagnostic for LLM Agent Societies](https://mlsanity.com/paper/2606.22203/) (2026)
- [Reason-Mediated Behavioral Models for Auditing LLM Social Simulators](https://mlsanity.com/paper/2607.24649/) (2026)
- [Benchmarking large language model agent societies against human behavioural distributions](https://mlsanity.com/paper/2608.28182/) (2026)
- [Why LLM Agents Favor Their Group: Stakes, Observed Norms, and Reputation](https://mlsanity.com/paper/2610.11008/) (2026)

## Abstract (arXiv)

Ashery et al. recently argue that large language models (LLMs), when paired to play a classic "naming game," spontaneously develop linguistic conventions reminiscent of human social norms. Here, we show that their results are better explained by data leakage: the models simply reproduce conventions they already encountered during pre-training. Despite the authors' mitigation measures, we provide multiple analyses demonstrating that the LLMs recognize the structure of the coordination game and recall its outcomes, rather than exhibit "emergent" conventions. Consequently, the observed behaviors are indistinguishable from memorization of the training corpus. We conclude by pointing to potential alternative strategies and reflecting more generally on the place of LLMs for social science models.