mlsanityData to 13 Oct 2026

Studying and improving reasoning in humans and machines

arXiv 2309.12485 ↗First posted 21 Sep 2023cs.CL cs.AI cs.LGcited by 14

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In the present study, we investigate and compare reasoning in large language models (LLM) and humans using a selection of cognitive psychology tools traditionally dedicated to the study of (bounded) rationality. To do so, we presented to human participants and an array of pretrained LLMs new variants of classical cognitive experiments, and cross-compared their performances. Our results showed that most of the included models presented reasoning errors akin to those frequently ascribed to error-prone, heuristic-based human reasoning. Notwithstanding this superficial similarity, an in-depth comparison between humans and LLMs indicated important differences with human-like reasoning, with models limitations disappearing almost entirely in more recent LLMs releases. Moreover, we show that while it is possible to devise strategies to induce better performance, humans and machines are not equally-responsive to the same prompting schemes. We conclude by discussing the epistemological implications and challenges of comparing human and machine behavior for both artificial intelligence and cognitive psychology.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 63 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

20 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Turning large language models into cognitive models6 Jun
  2. 2023Diminished Diversity-of-Thought in a Standard Large Language Model13 Feb

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,586
  10. 2023A Survey of Large Language Modelscited by 2,559
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  12. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 747
6 more
  1. 2022Using cognitive psychology to understand GPT-3cited by 395
  2. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  3. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  4. 2022Language Models as Agent Modelscited by 132
  5. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  6. 2020Understanding Human Intelligence through Human Limitationscited by 33

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024CogBench: a large language model walks into a psychology labcited by 61
  2. 2025The simulation of judgment in LLMscited by 22
  3. 2024Relative Value Biases in Large Language Modelscited by 6
  4. 2024Large Language Models are Biased Reinforcement Learnerscited by 5
  5. 2025How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognitioncited by 2
  6. 2026The role of System 1 and System 2 semantic memory structure in human and LLM biasescited by 1
  7. 2025Epitome: Pioneering an Experimental Platform for AI-Social Science Integrationcited by 0
  8. 2025Scaling Competence, Shrinking Reasoning: Cognitive Signatures in Language Model Learningcited by 0
  9. 2025MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Modelscited by 0
  10. 2026The Third Ambition: Artificial Intelligence and the Science of Human Behaviorcited by 0
  11. 2026How does Chain of Thought decompose complex tasks?cited by 0
  12. 2026LLMs struggle to simulate human belief updates in controlled environmentscited by 0
2 more
  1. 2026Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeedscited by 0
  2. 2026Assessing mentalization in humans and large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
  2. 2025Giving AI Personalities Leads to More Human-Like Reasoning
  3. 2026Reasoning Promotes Robustness in Theory of Mind Tasks
  4. 2025Beyond "Hallucinations": A Framework for Stable Human-AI Reasoning
  5. 2026Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
  6. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models
  7. 2024Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models
  8. 2025Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinking