mlsanityData to 7 Oct 2026

E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning

arXiv 2203.08480 ↗First posted 16 Mar 2022cs.CL cs.AIcited by 28

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The ability to recognize analogies is fundamental to human cognition. Existing benchmarks to test word analogy do not reveal the underneath process of analogical reasoning of neural models. Holding the belief that models capable of reasoning should be right for the right reasons, we propose a first-of-its-kind Explainable Knowledge-intensive Analogical Reasoning benchmark (E-KAR). Our benchmark consists of 1,655 (in Chinese) and 1,251 (in English) problems sourced from the Civil Service Exams, which require intensive background knowledge to solve. More importantly, we design a free-text explanation scheme to explain whether an analogy should be drawn, and manually annotate them for each and every question and candidate answer. Empirical results suggest that this benchmark is very challenging for some state-of-the-art models for both explanation generation and analogical question answering tasks, which invites further research in this area.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 43 of its 74 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

43 of 74 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
31 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  5. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  6. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  7. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  8. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  9. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  10. 2020Evaluation of Text Generation: A Surveycited by 275
  11. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  12. 2020Transformers as Soft Reasoners over Languagecited by 265
  13. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  14. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  15. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  16. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  17. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  18. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  19. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  20. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
  21. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  22. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  23. 2018Towards Understanding Linear Word Analogiescited by 57
  24. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  25. 2018Analogical Reasoning on Chinese Morphological and Semantic Relationscited by 50
  26. 2020Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answeringcited by 49
  27. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  28. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  29. 2019GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Levelcited by 17
  30. 2016Moving Beyond the Turing Test with the Allen AI Science Challengecited by 3
  31. 2017Compositional Approaches for Representing Relations Between Words: A Comparative Studycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 13; 2024: 6; 2025: 2; 2026: 2.

28 papers.

Sort:

By area:

  1. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  2. 2024Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Modelscited by 91
  3. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  4. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  5. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37
  6. 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
  7. 2023StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understandingcited by 26
  8. 2023In-Context Analogical Reasoning with Pre-Trained Language Modelscited by 22
  9. 2022Multimodal Analogical Reasoning over Knowledge Graphscited by 21
  10. 2022BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Modelscited by 20
  11. 2022This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polishcited by 13
  12. 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abductioncited by 11
16 more
  1. 2023OlaGPT: Empowering LLMs With Human-like Problem-Solving Abilitiescited by 10
  2. 2024ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogiescited by 8
  3. 2025Measuring Combinatorial Creativity in Large Language Models and Humanscited by 7
  4. 2026Emergent Analogical Reasoning in Transformerscited by 5
  5. 2022Harnessing Knowledge and Reasoning for Human-Like Natural Language Generation: A Brief Reviewcited by 3
  6. 2023FAME: Flexible, Scalable Analogy Mappings Enginecited by 3
  7. 2024Boosting Scientific Concepts Understanding: Can Analogy from Teacher Models Empower Student Models?cited by 3
  8. 2026MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generationcited by 2
  9. 2023A RelEntLess Benchmark for Modelling Graded Relations between Named Entitiescited by 1
  10. 2023RelBERT: Embedding Relations with Language Modelscited by 1
  11. 2023On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Modelscited by 1
  12. 2024Past Meets Present: Creating Historical Analogy with Large Language Modelscited by 1
  13. 2023Solving Hard Analogy Questions with Relation Embedding Chainscited by 0
  14. 2023Text Fact Transfercited by 0
  15. 2024SememeLM: A Sememe Knowledge Enhanced Method for Long-tail Relation Representationcited by 0
  16. 2025Investigating the Impact of Rationales for LLMs on Natural Language Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
  2. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies
  3. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance
  4. 2022Emergent Analogical Reasoning in Large Language Models
  5. 2025Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Research
  6. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
  7. 2023ARN: Analogical Reasoning on Narratives
  8. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Models