mlsanityData to 14 Oct 2026

Inductive reasoning in humans and large language models

arXiv 2306.06548 ↗First posted 11 Jun 2023cs.CL cs.AIcited by 24

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The impressive recent performance of large language models has led many to wonder to what extent they can serve as models of general intelligence or are similar to human cognition. We address this issue by applying GPT-3.5 and GPT-4 to a classic problem in human inductive reasoning known as property induction. Over two experiments, we elicit human judgments on a range of property induction tasks spanning multiple domains. Although GPT-3.5 struggles to capture many aspects of human behaviour, GPT-4 is much more successful: for the most part, its performance qualitatively matches that of humans, and the only notable exception is its failure to capture the phenomenon of premise non-monotonicity. Our work demonstrates that property induction allows for interesting comparisons between human and machine intelligence and provides two large datasets that can serve as benchmarks for future work in this vein.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 76 references that are in the map. First posted on arXiv 11 Jun 2023.

Builds on

20 of 76 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models24 May
  2. 2023Evaluating statistical language models as pragmatic reasoners1 May
  3. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  4. 2022Emergent Analogical Reasoning in Large Language Models19 Dec

Other references · 16

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  6. 2022Using cognitive psychology to understand GPT-3cited by 398
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  8. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  9. 2022Data Contamination: From Memorization to Exploitationcited by 236
  10. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  11. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  12. 2023Language Model Behavior: A Comprehensive Surveycited by 92
4 more
  1. 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 65
  2. 2020Word meaning in minds and machinescited by 51
  3. 2021Do language models learn typicality judgments from text?cited by 32
  4. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 11; 2025: 8; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  4. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  5. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  6. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  7. 2024Probing the limitations of multimodal language models for chemistry and materials researchcited by 32
  8. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  9. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  10. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  11. 2023Enhancing Genetic Improvement Mutations Using Large Language Modelscited by 13
  12. 2024An Evaluation of Estimative Uncertainty in Large Language Modelscited by 13
12 more
  1. 2024A Survey on Human-Centric LLMscited by 13
  2. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  3. 2025DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learningcited by 7
  4. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  5. 2024Philosophy of Cognitive Science in the Age of Deep Learningcited by 5
  6. 2024GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?cited by 5
  7. 2024Heuristic Reasoning in AI: Instrumental Use and Mimetic Absorptioncited by 3
  8. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  9. 2025On Language Models' Sensitivity to Suspicious Coincidencescited by 2
  10. 2025Predicate Renaming via Large Language Modelscited by 2
  11. 2025Iconicity in Large Language Modelscited by 1
  12. 2026Latent Taxonomies: How Property Induction Develops in Language Model Pre-trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Role of Model Prior in Real-World Inductive Reasoning
  2. 2026Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models
  3. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
  4. 2023Studying and improving reasoning in humans and machines
  5. 2025A Survey of Inductive Reasoning for Large Language Models
  6. 2025On LLM-Based Scientific Inductive Reasoning Beyond Equations
  7. 2026Thinking effort aligns between humans and reasoning models in abductive reasoning
  8. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models