# GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?

arXiv: https://arxiv.org/abs/2406.09464 · HTML: https://mlsanity.com/paper/2406.09464/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 30)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Cultural Bias and Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2311.14096/) (2023)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment](https://mlsanity.com/paper/2210.01478/) (2022)
- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [Large Language Models Produce Responses Perceived to be Empathic](https://mlsanity.com/paper/2403.18148/) (2024)
- [Inductive reasoning in humans and large language models](https://mlsanity.com/paper/2306.06548/) (2023)
- [What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theory](https://mlsanity.com/paper/2304.03612/) (2023)
- [Relational reasoning and generalization using non-symbolic neural networks](https://mlsanity.com/paper/2006.07968/) (2020)
- [Large Language Models are Capable of Offering Cognitive Reappraisal, if Guided](https://mlsanity.com/paper/2404.01288/) (2024)
- [Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models](https://mlsanity.com/paper/2310.14389/) (2023)
- [Exploring ChatGPT's Empathic Abilities](https://mlsanity.com/paper/2308.03527/) (2023)
- [Causal interventions expose implicit situation models for commonsense language understanding](https://mlsanity.com/paper/2306.03882/) (2023)

## Built on by (most cited first, 5 of 5)

- [GPT-4 Emulates Average-Human Emotional Cognition from a Third-Person Perspective](https://mlsanity.com/paper/2408.13718/) (2024)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [On the use of foundation models in cognitive science](https://mlsanity.com/paper/2608.07812/) (2026)

## Abstract (arXiv)

Large Language Models have taken the cognitive science world by storm. It is perhaps timely now to take stock of the various research paradigms that have been used to make scientific inferences about ``cognition" in these models or about human cognition. We review several emerging research paradigms -- GPT-ology, LLMs-as-computational-models, and ``silicon sampling" -- and review recent papers that have used LLMs under these paradigms. In doing so, we discuss their claims as well as challenges to scientific inference under these various paradigms. We highlight several outstanding issues about LLMs that have to be addressed to push our science forward: closed-source vs open-sourced models; (the lack of visibility of) training data; and reproducibility in LLM research, including forming conventions on new task ``hyperparameters" like instructions and prompts.