mlsanityData to 14 Oct 2026

Response: Emergent analogical reasoning in large language models

arXiv 2308.16118 ↗First posted 30 Aug 2023cs.CL cs.AIcited by 13

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In their recent Nature Human Behaviour paper, "Emergent analogical reasoning in large language models," (Webb, Holyoak, and Lu, 2023) the authors argue that "large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems." In this response, we provide counterexamples of the letter string analogies. In our tests, GPT-3 fails to solve simplest variations of the original tasks, whereas human performance remains consistently high across all modified versions. Zero-shot reasoning is an extraordinary claim that requires extraordinary evidence. We do not see that evidence in our experiments. To strengthen claims of humanlike reasoning such as zero-shot reasoning, it is important that the field develop approaches that rule out data memorization.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 2 of its 2 references that are in the map. First posted on arXiv 30 Aug 2023.

Builds on

2 of 2 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  2. 2022Emergent Analogical Reasoning in Large Language Models19 Dec

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 3; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Modelscited by 38
  2. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Modelscited by 31
  3. 2024LLMs as Models for Analogical Reasoningcited by 16
  4. 2024Evidence from counterfactual tasks supports emergent analogical reasoning in large language modelscited by 14
  5. 2025Do AI Models Perform Human-like Abstract Reasoning Across Modalities?cited by 10
  6. 2024Can Large Language Models generalize analogy solving like children can?cited by 9
  7. 2023How should the advent of large language models affect the practice of science?cited by 8
  8. 2024DeFine: Decision-Making with Analogical Reasoning over Factor Profilescited by 4
  9. 2025Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Researchcited by 3
  10. 2023Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applicationscited by 2
  11. 2025Genomic Next-Token Predictors are In-Context Learnerscited by 1
  12. 2026Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoningcited by 0
1 more
  1. 2026Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Gridcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023In-Context Analogical Reasoning with Pre-Trained Language Models
  2. 2024Inductive Linguistic Reasoning with Large Language Models
  3. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance
  4. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
  5. 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction
  6. 2024modeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
  7. 2023ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
  8. 2022Scientific and Creative Analogies in Pretrained Language Models