# The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories

arXiv: https://arxiv.org/abs/2501.12651 · HTML: https://mlsanity.com/paper/2501.12651/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 97 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 30 of 43)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [PersonaGym: Evaluating Persona Agents and LLMs](https://mlsanity.com/paper/2407.18416/) (2024)
- [Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voters](https://mlsanity.com/paper/2410.24190/) (2024)
- [Does Spatial Cognition Emerge in Frontier Models?](https://mlsanity.com/paper/2410.06468/) (2024)
- [CogBench: a large language model walks into a psychology lab](https://mlsanity.com/paper/2402.18225/) (2024)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Auxiliary task demands mask the capabilities of smaller language models](https://mlsanity.com/paper/2404.02418/) (2024)
- [Language models align with human judgments on key grammatical constructions](https://mlsanity.com/paper/2402.01676/) (2024)
- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [Centaur: a foundation model of human cognition](https://mlsanity.com/paper/2410.20268/) (2024)
- [Do language models learn typicality judgments from text?](https://mlsanity.com/paper/2105.02987/) (2021)
- [LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought](https://mlsanity.com/paper/2405.06705/) (2024)
- [KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models](https://mlsanity.com/paper/2407.17773/) (2024)
- [Evaluating Neural Language Models as Cognitive Models of Language Acquisition](https://mlsanity.com/paper/2310.20093/) (2023)

## Built on by (most cited first, 2 of 2)

- [Large Language Models as Psychological Simulators: A Methodological Guide](https://mlsanity.com/paper/2506.16702/) (2025)
- [Human-Like Anaphor Resolution in Large Language Models](https://mlsanity.com/paper/2608.05630/) (2026)

## Abstract (arXiv)

Many studies have evaluated the cognitive alignment of Pre-trained Language Models (PLMs), i.e., their correspondence to adult performance across a range of cognitive domains. Recently, the focus has expanded to the developmental alignment of these models: identifying phases during training where improvements in model performance track improvements in children's thinking over development. However, there are many challenges to the use of PLMs as cognitive science theories, including different architectures, different training data modalities and scales, and limited model interpretability. In this paper, we distill lessons learned from treating PLMs, not as engineering artifacts but as cognitive science and developmental science models. We review assumptions used by researchers to map measures of PLM performance to measures of human performance. We identify potential pitfalls of this approach to understanding human thinking, and we end by enumerating criteria for using PLMs as credible accounts of cognition and cognitive development.