mlsanityData to 14 Oct 2026

AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies

arXiv 2402.12370 ↗First posted 19 Feb 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Humans regularly engage in analogical thinking, relating personal experiences to current situations (X is analogous to Y because of Z). Analogical thinking allows humans to solve problems in creative ways, grasp difficult concepts, and articulate ideas more effectively. Can language models (LMs) do the same? To answer this question, we propose AnaloBench, a benchmark to determine analogical reasoning ability in LMs. Our benchmarking approach focuses on aspects of this ability that are common among humans: (i) recalling related experiences from a large amount of information, and (ii) applying analogical reasoning to complex and lengthy scenarios. We test a broad collection of proprietary models (e.g., GPT family, Claude V2) and open source models such as LLaMA2. As in prior results, scaling up LMs results in some performance boosts. Surprisingly, scale offers minimal gains when, (i) analogies involve lengthy scenarios, or (ii) recalling relevant scenarios from a large pool of information, a process analogous to finding a needle in a haystack. We hope these observations encourage further research in this field.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 31 of its 76 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

31 of 76 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  8. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  9. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  11. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  12. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
19 more
  1. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  2. 2023Large Language Models as Analogical Reasonerscited by 107
  3. 2018Towards Understanding Linear Word Analogiescited by 57
  4. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  5. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  6. 2017Accelerating Innovation Through Analogy Miningcited by 32
  7. 2022Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPTcited by 27
  8. 2023StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understandingcited by 26
  9. 2022Life is a Circus and We are the Clowns: Automatically Finding Analogies between Situations and Processescited by 22
  10. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
  11. 2015VISALOGY: Answering Visual Analogy Questionscited by 17
  12. 2023ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Basecited by 16
  13. 2021ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understandingcited by 14
  14. 2023Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphorscited by 12
  15. 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abductioncited by 11
  16. 2021A Neural Approach for Detecting Morphological Analogiescited by 6
  17. 2018Textual Analogy Parsing: What's Shared and What's Compared among Analogous Factscited by 5
  18. 2023Solving morphological analogies: from retrieval to generationcited by 3
  19. 2024Reframing Tax Law Entailment as Analogical Reasoningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 7.

13 papers.

Sort:

By area:

  1. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  2. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 40
  3. 2025LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoningcited by 12
  4. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  5. 2024Boosting Scientific Concepts Understanding: Can Analogy from Teacher Models Empower Student Models?cited by 3
  6. 2024Past Meets Present: Creating Historical Analogy with Large Language Modelscited by 1
  7. 2026Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Gridcited by 0
  8. 2026VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Imagescited by 0
  9. 2026Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generationcited by 0
  10. 2026Analogical Deep Research: Retrieving and Integrating Historical Analogies for Foresight Analysiscited by 0
  11. 2026On the Diversity of Analogy Making in Large Language Modelscited by 0
  12. 2026RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literaturecited by 0
1 more
  1. 2026PRISM: A Category-Theoretic Framework for Measuring and Refining Multimodal Analogiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
  2. 2025Modeling Understanding of Story-Based Analogies Using Large Language Models
  3. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Models
  4. 2024LLMs as Models for Analogical Reasoning
  5. 2025The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
  6. 2022Emergent Analogical Reasoning in Large Language Models
  7. 2024Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?
  8. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance