# Response: Emergent analogical reasoning in large language models

arXiv: https://arxiv.org/abs/2308.16118 · HTML: https://mlsanity.com/paper/2308.16118/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 2 of its 2 references that are in the map.
- First posted on arXiv 30 Aug 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 2 of 2)

- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)

## Built on by (most cited first, 13 of 13)

- [Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2402.08955/) (2024)
- [Evaluating the Robustness of Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2411.14215/) (2024)
- [LLMs as Models for Analogical Reasoning](https://mlsanity.com/paper/2406.13803/) (2024)
- [Evidence from counterfactual tasks supports emergent analogical reasoning in large language models](https://mlsanity.com/paper/2404.13070/) (2024)
- [Do AI Models Perform Human-like Abstract Reasoning Across Modalities?](https://mlsanity.com/paper/2510.02125/) (2025)
- [Can Large Language Models generalize analogy solving like children can?](https://mlsanity.com/paper/2411.02348/) (2024)
- [How should the advent of large language models affect the practice of science?](https://mlsanity.com/paper/2312.03759/) (2023)
- [DeFine: Decision-Making with Analogical Reasoning over Factor Profiles](https://mlsanity.com/paper/2410.01772/) (2024)
- [Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Research](https://mlsanity.com/paper/2509.09381/) (2025)
- [Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applications](https://mlsanity.com/paper/2311.15716/) (2023)
- [Genomic Next-Token Predictors are In-Context Learners](https://mlsanity.com/paper/2511.12797/) (2025)
- [Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoning](https://mlsanity.com/paper/2604.06501/) (2026)
- [Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Grid](https://mlsanity.com/paper/2605.01359/) (2026)

## Abstract (arXiv)

In their recent Nature Human Behaviour paper, "Emergent analogical reasoning in large language models," (Webb, Holyoak, and Lu, 2023) the authors argue that "large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems." In this response, we provide counterexamples of the letter string analogies. In our tests, GPT-3 fails to solve simplest variations of the original tasks, whereas human performance remains consistently high across all modified versions. Zero-shot reasoning is an extraordinary claim that requires extraordinary evidence. We do not see that evidence in our experiments. To strengthen claims of humanlike reasoning such as zero-shot reasoning, it is important that the field develop approaches that rule out data memorization.