mlsanityData to 14 Oct 2026

Evidence from counterfactual tasks supports emergent analogical reasoning in large language models

arXiv 2404.13070 ↗First posted 14 Apr 2024cs.CL cs.AIcited by 14

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We recently reported evidence that large language models are capable of solving a wide range of text-based analogy problems in a zero-shot manner, indicating the presence of an emergent capacity for analogical reasoning. Two recent commentaries have challenged these results, citing evidence from so-called `counterfactual' tasks in which the standard sequence of the alphabet is arbitrarily permuted so as to decrease similarity with materials that may have been present in the language model's training data. Here, we reply to these critiques, clarifying some misunderstandings about the test materials used in our original work, and presenting evidence that language models are also capable of generalizing to these new counterfactual task variants.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 17 of its 32 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

17 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models14 Feb
  2. 2023Response: Emergent analogical reasoning in large language models30 Aug
  3. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  4. 2022Emergent Analogical Reasoning in Large Language Models19 Dec

Other references · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022In-context Learning and Induction Headscited by 964
  4. 2022Using cognitive psychology to understand GPT-3cited by 398
  5. 2023In-Context Learning Creates Task Vectorscited by 312
  6. 2023Function Vectors in Large Language Modelscited by 308
  7. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  8. 2023How do Language Models Bind Entities in Context?cited by 108
  9. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  10. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  11. 2023The Relational Bottleneck as an Inductive Bias for Efficient Abstractioncited by 27
  12. 2021Probabilistic Analogical Mapping with Semantic Relation Networkscited by 14
1 more
  1. 2022Zero-shot visual reasoning through probabilistic analogical mappingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 5.

14 papers.

Sort:

By area:

  1. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  2. 2024The broader spectrum of in-context learningcited by 35
  3. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  4. 2024LLMs as Models for Analogical Reasoningcited by 16
  5. 2024Can Large Language Models generalize analogy solving like children can?cited by 9
  6. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  7. 2024Anthropocentric bias in language model evaluationcited by 4
  8. 2025Reasoning about Affordances: Causal and Compositional Reasoning in LLMscited by 1
  9. 2025Genomic Next-Token Predictors are In-Context Learnerscited by 1
  10. 2026Relational Knowledge Distillation Using Fine-tuned Function Vectorscited by 0
  11. 2026Emergent Structured Representations Support Flexible In-Context Inference in Large Language Modelscited by 0
  12. 2026Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarioscited by 0
2 more
  1. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0
  2. 2026On the use of foundation models in cognitive sciencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Models
  2. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios
  3. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
  4. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
  5. 2022Counterfactual reasoning: Do language models need world knowledge for causal understanding?
  6. 2023What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
  7. 2024Inductive Linguistic Reasoning with Large Language Models
  8. 2025Modeling Understanding of Story-Based Analogies Using Large Language Models