mlsanityData to 7 Oct 2026

Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples

arXiv 2305.15269 ↗First posted 24 May 2023cs.CL cs.AIcited by 84

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Given the intractably large size of the space of proofs, any model that is capable of general deductive reasoning must generalize to proofs of greater complexity. Recent studies have shown that large language models (LLMs) possess some abstract deductive reasoning ability given chain-of-thought prompts. However, they have primarily been tested on proofs using modus ponens or of a specific size, and from the same distribution as the in-context examples. To measure the general deductive reasoning ability of LLMs, we test on a broad set of deduction rules and measure their ability to generalize to more complex proofs from simpler demonstrations from multiple angles: depth-, width-, and compositional generalization. To facilitate systematic exploration, we construct a new synthetic and programmable reasoning dataset that enables control over deduction rules and proof complexity. Our experiments on four LLMs of various sizes and training objectives show that they are able to generalize to compositional proofs. However, they have difficulty generalizing to longer proofs, and they require explicit demonstrations to produce hypothetical subproofs, specifically in proof by cases and proof by contradiction.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 32 of its 41 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

32 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023How Do In-Context Examples Affect Compositional Generalization?8 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Language20 Dec
  4. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  5. 2022Evaluating Step-by-Step Reasoning through Symbolic Verification16 Dec
  6. 2022Measuring and Narrowing the Compositionality Gap in Language Models7 Oct
  7. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  8. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  9. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning25 May
  10. 2022On the Paradox of Learning to Reason from Data23 May
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  11. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  12. 2022Exploring Length Generalization in Large Language Modelscited by 204
9 more
  1. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  2. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  3. 2022Inverse scaling can become U-shapedcited by 76
  4. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  5. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  6. 2020INT: An Inequality Benchmark for Evaluating Generalization in Theorem Provingcited by 44
  7. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  8. 2022On the Compositional Generalization Gap of In-Context Learningcited by 24
  9. 2023Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 27; 2025: 30; 2026: 10.

84 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  3. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  4. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  5. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  6. 2025BIG-Bench Extra Hardcited by 123
  7. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  8. 2023On Generative Agents in Recommendationcited by 88
  9. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  10. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  11. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  12. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
38 more
  1. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  2. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
  3. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  4. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  5. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  6. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  7. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  8. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  9. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27
  10. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26
  11. 2024How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendationcited by 25
  12. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  13. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  14. 2024OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Modelscited by 23
  15. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  16. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Modelscited by 20
  17. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  18. 2024IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abductioncited by 20
  19. 2024UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorizationcited by 17
  20. 2025FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generationcited by 17
  21. 2024Can Transformers Reason Logically? A Study in SAT Solvingcited by 16
  22. 2025BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discoverycited by 15
  23. 2023GLoRE: Evaluating Logical Reasoning of Large Language Modelscited by 13
  24. 2024A Survey on Human-Centric LLMscited by 13
  25. 2023Evaluating the Deductive Competence of Large Language Modelscited by 12
  26. 2024A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systemscited by 12
  27. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  28. 2025LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoningcited by 12
  29. 2025Task Generalization With AutoRegressive Compositional Structure: Can Learning From D Tasks Generalize to DT Tasks?cited by 11
  30. 2024Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?cited by 10
  31. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  32. 2024Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verificationcited by 9
  33. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  34. 2023Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysiscited by 6
  35. 2023Multi-Operational Mathematical Derivations in Latent Spacecited by 6
  36. 2025Enhancing Transformers for Generalizable First-Order Logical Entailmentcited by 6
  37. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  38. 2025Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Provingcited by 6

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models
  2. 2025Are Language Models Efficient Reasoners? A Perspective from Logic Programming
  3. 2025Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Models
  4. 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation
  5. 2026Large Language Model for OWL Proofs
  6. 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models
  7. 2025ProofCompass: Enhancing Specialized Provers with LLM Guidance
  8. 2025Safe: Enhancing Mathematical Reasoning in Large Language Models via Retrospective Step-aware Formal Verification