# LLM Cognitive Judgements Differ From Human

arXiv: https://arxiv.org/abs/2307.11787 · HTML: https://mlsanity.com/paper/2307.11787/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Do large language models resemble humans in language use?](https://mlsanity.com/paper/2303.08014/) (2023)
- [Does ChatGPT have Theory of Mind?](https://mlsanity.com/paper/2305.14020/) (2023)
- [Evaluating statistical language models as pragmatic reasoners](https://mlsanity.com/paper/2305.01020/) (2023)
- [Does Conceptual Representation Require Embodiment? Insights From Large Language Models](https://mlsanity.com/paper/2305.19103/) (2023)

## Built on by (most cited first, 8 of 8)

- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [(Ir)rationality and Cognitive Biases in Large Language Models](https://mlsanity.com/paper/2402.09193/) (2024)
- [Using Artificial Populations to Study Psychological Phenomena in Neural Models](https://mlsanity.com/paper/2308.08032/) (2023)
- [Large Language Model Recall Uncertainty is Modulated by the Fan Effect](https://mlsanity.com/paper/2407.06349/) (2024)
- [Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji](https://mlsanity.com/paper/2504.02116/) (2025)
- [From Words to Worlds: Compositionality for Cognitive Architectures](https://mlsanity.com/paper/2407.13419/) (2024)
- [Meaningless is better: hashing bias-inducing words in LLM prompts improves performance in logical reasoning and statistical learning](https://mlsanity.com/paper/2411.17304/) (2024)
- [The Anatomy of Speech Persuasion: Linguistic Shifts in LLM-Modified Speeches](https://mlsanity.com/paper/2506.18621/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have lately been on the spotlight of researchers, businesses, and consumers alike. While the linguistic capabilities of such models have been studied extensively, there is growing interest in investigating them as cognitive subjects. In the present work I examine GPT-3 and ChatGPT capabilities on an limited-data inductive reasoning task from the cognitive science literature. The results suggest that these models' cognitive judgements are not human-like.