# Studying and improving reasoning in humans and machines

arXiv: https://arxiv.org/abs/2309.12485 · HTML: https://mlsanity.com/paper/2309.12485/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 63 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Language Models as Agent Models](https://mlsanity.com/paper/2212.01681/) (2022)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Turning large language models into cognitive models](https://mlsanity.com/paper/2306.03917/) (2023)
- [Diminished Diversity-of-Thought in a Standard Large Language Model](https://mlsanity.com/paper/2302.07267/) (2023)
- [Understanding Human Intelligence through Human Limitations](https://mlsanity.com/paper/2009.14050/) (2020)

## Built on by (most cited first, 14 of 14)

- [CogBench: a large language model walks into a psychology lab](https://mlsanity.com/paper/2402.18225/) (2024)
- [The simulation of judgment in LLMs](https://mlsanity.com/paper/2502.04426/) (2025)
- [Relative Value Biases in Large Language Models](https://mlsanity.com/paper/2401.14530/) (2024)
- [Large Language Models are Biased Reinforcement Learners](https://mlsanity.com/paper/2405.11422/) (2024)
- [How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognition](https://mlsanity.com/paper/2503.00330/) (2025)
- [MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models](https://mlsanity.com/paper/2512.05530/) (2025)
- [The role of System 1 and System 2 semantic memory structure in human and LLM biases](https://mlsanity.com/paper/2604.12816/) (2026)
- [Assessing mentalization in humans and large language models](https://mlsanity.com/paper/2608.26291/) (2026)
- [Epitome: Pioneering an Experimental Platform for AI-Social Science Integration](https://mlsanity.com/paper/2507.01061/) (2025)
- [Scaling Competence, Shrinking Reasoning: Cognitive Signatures in Language Model Learning](https://mlsanity.com/paper/2511.21743/) (2025)
- [The Third Ambition: Artificial Intelligence and the Science of Human Behavior](https://mlsanity.com/paper/2603.07329/) (2026)
- [How does Chain of Thought decompose complex tasks?](https://mlsanity.com/paper/2604.08872/) (2026)
- [LLMs struggle to simulate human belief updates in controlled environments](https://mlsanity.com/paper/2607.28347/) (2026)
- [Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds](https://mlsanity.com/paper/2607.28908/) (2026)

## Abstract (arXiv)

In the present study, we investigate and compare reasoning in large language models (LLM) and humans using a selection of cognitive psychology tools traditionally dedicated to the study of (bounded) rationality. To do so, we presented to human participants and an array of pretrained LLMs new variants of classical cognitive experiments, and cross-compared their performances. Our results showed that most of the included models presented reasoning errors akin to those frequently ascribed to error-prone, heuristic-based human reasoning. Notwithstanding this superficial similarity, an in-depth comparison between humans and LLMs indicated important differences with human-like reasoning, with models limitations disappearing almost entirely in more recent LLMs releases. Moreover, we show that while it is possible to devise strategies to induce better performance, humans and machines are not equally-responsive to the same prompting schemes. We conclude by discussing the epistemological implications and challenges of comparing human and machine behavior for both artificial intelligence and cognitive psychology.