mlsanityData to 7 Oct 2026

Language Models as Inductive Reasoners

arXiv 2212.10923 ↗First posted 21 Dec 2022cs.CL cs.AIcited by 32

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Inductive reasoning is a core component of human intelligence. In the past research of inductive reasoning within computer science, formal language is used as representations of knowledge (facts and rules, more specifically). However, formal language can cause systematic problems for inductive reasoning such as disability of handling raw input such as natural language, sensitiveness to mislabeled data, and incapacity to handle ambiguous input. To this end, we propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. New automatic metrics are also proposed and analysed for the evaluation of this task. With DEER, we investigate a modern approach for inductive reasoning where we use natural language as representation for knowledge instead of formal language and use pretrained language models as ''reasoners''. Moreover, we provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts. We also propose a new framework drawing insights from philosophy literature for this task, which we show in the experiment section that surpasses baselines in both automatic and human evaluations. We discuss about our future perspectives for inductive reasoning in Section 7. Dataset and code are available at https://github.com/ZonglinY/Inductive_Reasoning.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 67 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

24 of 67 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  7. 2019Abductive Commonsense Reasoningcited by 300
  8. 2020Transformers as Soft Reasoners over Languagecited by 265
  9. 2019Inductive Relation Prediction by Subgraph Reasoningcited by 237
  10. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  11. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  12. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
12 more
  1. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  2. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  3. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  4. 2021Inductive logic programming at 30cited by 46
  5. 2021Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Trainingcited by 41
  6. 2020Abductive Knowledge Induction From Raw Datacited by 38
  7. 2022Knowledge Base Question Answering by Case-based Reasoning over Subgraphscited by 31
  8. 2020Improving Event Duration Prediction via Time-aware Pre-trainingcited by 13
  9. 2022Natural Language Deduction with Incomplete Informationcited by 12
  10. 2021Learning Symbolic Rules for Reasoning in Quasi-Natural Languagecited by 9
  11. 2021Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledgecited by 8
  12. 2022A Property Induction Framework for Neural Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 9; 2025: 9; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  2. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  3. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  4. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  5. 2024Hypothesis Generation with Large Language Modelscited by 68
  6. 2024MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypothesescited by 68
  7. 2023Large Language Models can Learn Rulescited by 66
  8. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 61
  9. 2023Natural Language Reasoning, A Surveycited by 61
  10. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 52
  11. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approachescited by 43
  12. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 40
20 more
  1. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 40
  2. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  3. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  4. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Surveycited by 32
  5. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  6. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  7. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  8. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  9. 2023Large Language Model Programscited by 11
  10. 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 10
  11. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Textcited by 5
  12. 2025Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Taskcited by 4
  13. 2026Generating Literature-Driven Scientific Theories at Scalecited by 3
  14. 2025On Language Models' Sensitivity to Suspicious Coincidencescited by 2
  15. 2025Predicate Renaming via Large Language Modelscited by 2
  16. 2024PyGen: A Collaborative Human-AI Approach to Python Package Creationcited by 1
  17. 2025Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interactioncited by 1
  18. 2025Enhancing Computational Cognitive Architectures with LLMs: A Case Studycited by 0
  19. 2026CoGe-GCD: Reframing Generalized Category Discovery with Compositional Generalizationcited by 0
  20. 2026Generalized Biomedicine Discoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
  2. 2023Hypothesis Search: Inductive Reasoning with Language Models
  3. 2025A Survey of Inductive Reasoning for Large Language Models
  4. 2025InductionBench: LLMs Fail in the Simplest Complexity Class
  5. 2024Generating Diverse Hypotheses for Inductive Reasoning
  6. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
  7. 2025On LLM-Based Scientific Inductive Reasoning Beyond Equations
  8. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models