mlsanityData to 14 Oct 2026

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs

arXiv 2402.11442 ↗First posted 18 Feb 2024cs.CLcited by 28

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have achieved impressive human-like performance across various reasoning tasks. However, their mastery of underlying inferential rules still falls short of human capabilities. To investigate this, we propose a logic scaffolding inferential rule generation framework, to construct an inferential rule base, ULogic, comprising both primitive and compositional rules across five domains. Our analysis of GPT-series models over a rule subset reveals significant gaps in LLMs' logic understanding compared to human performance, especially in compositional and structural complex rules with certain bias patterns. We further distill these rules into a smaller-scale inference engine for flexible rule generation and enhancing downstream reasoning. Through a multi-judger evaluation, our inference engine proves effective in generating accurate, complex and abstract conclusions and premises, and improve various commonsense reasoning tasks. Overall, our work sheds light on LLMs' limitations in grasping inferential rule and suggests ways to enhance their logical reasoning abilities~footnoteCode and data are available at urlhttps://github.com/SiyuanWangw/ULogic..

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 37 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

23 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers23 Oct

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  8. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  9. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  10. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  11. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  12. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
10 more
  1. 2020RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphscited by 118
  2. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  3. 2023Large Language Models can Learn Rulescited by 69
  4. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  5. 2023Can LLMs Follow Simple Rules?cited by 46
  6. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43
  7. 2019Learn to Explain Efficiently via Neural Logic Inductive Learningcited by 38
  8. 2021Neuro-Symbolic Inductive Logic Programming with Logical Neural Networkscited by 36
  9. 2021Neuro-Symbolic Forward Reasoningcited by 16
  10. 2022Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Promptscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 8; 2026: 9.

28 papers.

Sort:

By area:

  1. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  2. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  3. 2026Large Language Model Reasoning Failurescited by 21
  4. 2024Symbolic Working Memory Enhances Language Models for Complex Rule Applicationcited by 17
  5. 2024LogicPro: Improving Complex Logical Reasoning via Program-Guided Learningcited by 14
  6. 2024Can Large Language Models Grasp Legal Theories? Enhance Legal Reasoning with Insights from Multi-Agent Collaborationcited by 14
  7. 2024RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answeringcited by 13
  8. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  9. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7
  10. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  11. 2024CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Modelscited by 4
  12. 2024Can Large Language Models Understand DL-Lite Ontologies? An Empirical Studycited by 4
16 more
  1. 2025Towards Quantifying Commonsense Reasoning with Mechanistic Insightscited by 4
  2. 2025Understanding Subword Compositionality of Large Language Modelscited by 3
  3. 2025Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modelingcited by 2
  4. 2025RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learningcited by 2
  5. 2026How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge? A Software Engineering Frameworkcited by 2
  6. 2024From Words to Worlds: Compositionality for Cognitive Architecturescited by 1
  7. 2024Exploiting LLMs' Reasoning Capability to Infer Implicit Concepts in Legal Information Retrievalcited by 1
  8. 2025Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)cited by 1
  9. 2025INSIGHT: An Interpretable Neural Vision-Language Framework for Reasoning of Generative Artifactscited by 1
  10. 2026CascadeMind at SemEval-2026 Task 4: A Hybrid Neuro-Symbolic Cascade for Narrative Similaritycited by 1
  11. 2026LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verificationcited by 1
  12. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playingcited by 0
  13. 2026ttda704 at SemEval-2026 Task 4: Modeling Narrative Structures via Pseudonymization and Multi-View Sentence Alignmentcited by 0
  14. 2026LLMs Learn Better In-Context from Rules than from Examplescited by 0
  15. 2026Detecting Soft Errors in Parallel Software with LLM-tuned Instruction Duplicationcited by 0
  16. 2026Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios
  2. 2024Do Large Language Models Understand Logic or Just Mimick Context?
  3. 2025LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning
  4. 2024RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
  5. 2025Guiding Reasoning in Small Language Models with LLM Assistance
  6. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Study
  7. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  8. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models