# Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

arXiv: https://arxiv.org/abs/2505.16164 · HTML: https://mlsanity.com/paper/2505.16164/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 29 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 4 of 4)

- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Network Filtering for Big Data: Triangulated Maximally Filtered Graph](https://mlsanity.com/paper/1505.02445/) (2015)
- [Semantic features of object concepts generated with GPT-3](https://mlsanity.com/paper/2202.03753/) (2022)
- [Cognitive networks highlight differences and similarities in the STEM mindsets of human and LLM-simulated trainees, experts and academics](https://mlsanity.com/paper/2502.19529/) (2025)

## Built on by (most cited first, 4 of 4)

- [Can GPT replace human raters? Validity and reliability of machine-generated norms for metaphors](https://mlsanity.com/paper/2512.12444/) (2025)
- [Comparing Semantic Navigation in Humans and Large Language Models using Natural Language Processing](https://mlsanity.com/paper/2607.12195/) (2026)
- [Focus particles and scalar inferences across humans and language models](https://mlsanity.com/paper/2608.08227/) (2026)
- [AI Models Can Predict and Collaboratively Modulate Human Memory Search](https://mlsanity.com/paper/2608.26152/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks, but their ability to simulate human behavioral variability remains unclear. This study examines whether LLMs can approximate individual differences in the phonemic fluency task, where participants generate words beginning with a target letter. We evaluated 34 distinct models across 45 configurations from major closed-source and open-source providers, and compared outputs to responses from 106 human participants. While some models, especially Claude 3.7 Sonnet, approximated human averages and lexical preferences, none reproduced the scope of human variability. LLM outputs were consistently less diverse, with newer models and thinking-enabled modes often reducing rather than increasing variability. Network analysis further revealed fundamental differences in retrieval structure between humans and the most human-like model. Ensemble simulations combining outputs from diverse models also failed to recover human-level diversity, likely due to high vocabulary overlap across models. These results highlight key limitations in using LLMs to simulate human cognition and behavior.