mlsanityData to 7 Oct 2026

Large Language Models can Learn Rules

arXiv 2310.07064 ↗First posted 10 Oct 2023cs.AI cs.CLcited by 66

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When prompted with a few examples and intermediate steps, large language models (LLMs) have demonstrated impressive performance in various reasoning tasks. However, prompting methods that rely on implicit knowledge in an LLM often generate incorrect answers when the implicit knowledge is wrong or inconsistent with the task. To tackle this problem, we present Hypotheses-to-Theories (HtT), a framework that learns a rule library for reasoning with LLMs. HtT contains two stages, an induction stage and a deduction stage. In the induction stage, an LLM is first asked to generate and verify rules over a set of training examples. Rules that appear and lead to correct answers sufficiently often are collected to form a rule library. In the deduction stage, the LLM is then prompted to employ the learned rule library to perform reasoning to answer test questions. Experiments on relational reasoning, numerical reasoning and concept learning problems show that HtT improves existing prompting methods, with an absolute gain of 10-30% in accuracy. The learned rules are also transferable to different models and to different forms of the same problem.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 64 of its 86 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

64 of 86 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2023Let's Verify Step by Stepcited by 3,598
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  13. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  14. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  15. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  16. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  17. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  18. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  19. 2014Memory Networkscited by 354
  20. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  21. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  22. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  23. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  24. 2023How Language Model Hallucinations Can Snowballcited by 310
  25. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  26. 2023Large Language Models as Tool Makerscited by 293
  27. 2020Transformers as Soft Reasoners over Languagecited by 265
  28. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  29. 2019Inductive Relation Prediction by Subgraph Reasoningcited by 237
  30. 2022TALM: Tool Augmented Language Modelscited by 197
  31. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  32. 2017End-to-End Differentiable Provingcited by 159
  33. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  34. 2020RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphscited by 117
  35. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 116
  36. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  37. 2015Neural Programmer-Interpreterscited by 96
  38. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 37; 2025: 24; 2026: 4.

66 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  5. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 117
  6. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  7. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 92
  8. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  9. 2024Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Modelscited by 55
  10. 2024AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learningcited by 52
  11. 2023Can LLMs Follow Simple Rules?cited by 45
  12. 2025AlphaAgent: LLM-Driven Alpha Mining with Regularized Exploration to Counteract Alpha Decaycited by 36
38 more
  1. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 35
  2. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  3. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  4. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  5. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
  6. 2024IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abductioncited by 20
  7. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 20
  8. 2024MetaReflection: Learning Instructions for Language Agents using Past Reflectionscited by 19
  9. 2024Symbolic Working Memory Enhances Language Models for Complex Rule Applicationcited by 17
  10. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  11. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  12. 2024Integrating Intent Understanding and Optimal Behavior Planning for Behavior Tree Generation from Human Instructionscited by 12
  13. 2025AlphaOPT: Formulating Optimization Programs with Self-Improving LLM Experience Librarycited by 12
  14. 2024RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Modelscited by 11
  15. 2025LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discoverycited by 11
  16. 2024Enhancing Agent Learning through World Dynamics Modelingcited by 9
  17. 2025Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Modelscited by 9
  18. 2024ItD: Large Language Models Can Teach Themselves Induction through Deductioncited by 8
  19. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  20. 2024RAGProbe: An Automated Approach for Evaluating RAG Applicationscited by 7
  21. 2025DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learningcited by 7
  22. 2024Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialoguescited by 6
  23. 2025Affordable AI Assistants with Knowledge Graph of Thoughtscited by 6
  24. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  25. 2024DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMscited by 5
  26. 2024TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidancecited by 5
  27. 2025WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentscited by 5
  28. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Modelscited by 5
  29. 2024ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Modelcited by 4
  30. 2024RuAG: Learned-rule-augmented Generation for Large Language Modelscited by 4
  31. 2025Large Language Models for Extrapolative Modeling of Manufacturing Processescited by 4
  32. 2024Brain-Inspired Two-Stage Approach: Enhancing Mathematical Reasoning by Imitating Human Thought Processescited by 3
  33. 2024HALO: Hallucination Analysis and Learning Optimization to Empower LLMs with Retrieval-Augmented Context for Guided Clinical Decision Makingcited by 3
  34. 2025Rule-Guided Feedback: Enhancing Reasoning by Enforcing Rule Adherence in Large Language Modelscited by 3
  35. 2025Large Language and Reasoning Models are Shallow Disjunctive Reasonerscited by 3
  36. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
  37. 2024Intelligent OPC Engineer Assistant for Semiconductor Manufacturingcited by 2
  38. 2024Inductive Linguistic Reasoning with Large Language Modelscited by 2

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024General Purpose Verification for Chain of Thought Prompting
  2. 2023Hypothesis Search: Inductive Reasoning with Language Models
  3. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  4. 2025Reasoning Capabilities and Invariability of Large Language Models
  5. 2025Reasoning Can Hurt the Inductive Abilities of Large Language Models
  6. 2022Large Language Models Are Reasoning Teachers
  7. 2024Do Large Language Models Understand Logic or Just Mimick Context?
  8. 2023Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning