mlsanityData to 7 Oct 2026

Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models

arXiv 2402.08955 ↗First posted 14 Feb 2024cs.AI cs.CLcited by 38

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, it has been debated whether they are actually performing humanlike abstract reasoning or instead employing less general processes that rely on similarity to what has been seen in their training data. Here we investigate the generality of analogy-making abilities previously claimed for LLMs (Webb, Holyoak, & Lu, 2023). We take one set of analogy problems used to evaluate LLMs and create a set of "counterfactual" variants-versions that test the same abstract reasoning abilities but that are likely dissimilar from any pre-training data. We test humans and three GPT models on both the original and counterfactual problems, and show that, while the performance of humans remains high for all the problems, the GPT models' performance declines sharply on the counterfactual set. This work provides evidence that, despite previously reported successes of LLMs on analogical reasoning, these models lack the robustness and generality of human analogy-making.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 16 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

8 of 16 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Response: Emergent analogical reasoning in large language models30 Aug
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  3. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  4. 2022Emergent Analogical Reasoning in Large Language Models19 Dec

Other references · 4

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  4. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 12; 2026: 9.

38 papers.

Sort:

By area:

  1. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 67
  2. 2024Not All LLM Reasoners Are Created Equalcited by 29
  3. 2024LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actionscited by 21
  4. 2024LLMs as Models for Analogical Reasoningcited by 16
  5. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  6. 2025Achieving Time Series Reasoning Requires Rethinking Model Design, Tasks Formulation, and Evaluationcited by 15
  7. 2024Evidence from counterfactual tasks supports emergent analogical reasoning in large language modelscited by 14
  8. 2024Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verificationcited by 9
  9. 2024Towards Learning Abductive Reasoning using VSA Distributed Representationscited by 9
  10. 2024Can Large Language Models generalize analogy solving like children can?cited by 9
  11. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  12. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
26 more
  1. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  2. 2025Revealing emergent human-like conceptual representations from language predictioncited by 5
  3. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 5
  4. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  5. 2024Anthropocentric bias in language model evaluationcited by 4
  6. 2026A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilitiescited by 4
  7. 2024DeFine: Decision-Making with Analogical Reasoning over Factor Profilescited by 3
  8. 2025Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Researchcited by 3
  9. 2024Aligning Generalisation Between Humans and Machinescited by 2
  10. 2025Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worldscited by 2
  11. 2025Empowerment Gain and Causal Model Construction: Children and adults are sensitive to controllability and variability in their causal interventionscited by 2
  12. 2024Visual Reasoning in Object-Centric Deep Neural Networks: A Comparative Cognition Approachcited by 1
  13. 2024Large Language Models, and LLM-Based Agents, Should Be Used to Enhance the Digital Public Spherecited by 1
  14. 2025DeduCE: Deductive Consistency as a Framework to Evaluate LLM Reasoningcited by 1
  15. 2024Analogical Reasoning Within a Conceptual Hyperspacecited by 0
  16. 2025LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoningcited by 0
  17. 2025AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguitycited by 0
  18. 2025DEVAL: A Framework for Evaluating and Improving the Derivation Capability of Large Language Modelscited by 0
  19. 2026Relational Knowledge Distillation Using Fine-tuned Function Vectorscited by 0
  20. 2026Emergent Structured Representations Support Flexible In-Context Inference in Large Language Modelscited by 0
  21. 2026Large Language Models provide support for the parallelogram theory of analogycited by 0
  22. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  23. 2026Normative Robustness as a Frontier for Non-Verifiable Reasoning in LLMscited by 0
  24. 2026Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoningcited by 0
  25. 2026The Metanym Game: An LLM Benchmark Without Ground Truth That Rises With the Models It Measurescited by 0
  26. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Models
  2. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
  3. 2025Modeling Understanding of Story-Based Analogies Using Large Language Models
  4. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance
  5. 2025The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
  6. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  7. 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction
  8. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios