mlsanityData to 7 Oct 2026

Examining the Emergence of Deductive Reasoning in Generative Language Models

arXiv 2306.01009 ↗First posted 31 May 2023cs.CL cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We conduct a preliminary inquiry into the ability of generative transformer models to deductively reason from premises provided. We observe notable differences in the performance of models coming from different training setups and find that the deductive reasoning ability increases with scale. Further, we discover that the performance generally does not decrease with the length of the deductive chain needed to reach the conclusion, with the exception of OpenAI GPT-3 and GPT-3.5 models. Our study considers a wide variety of transformer-decoder models, ranging from 117 million to 175 billion parameters in size.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 12 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

10 of 12 references.

References · 10

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  4. 2020Transformers as Soft Reasoners over Languagecited by 265
  5. 2021Entailment as Few-Shot Learnercited by 122
  6. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  7. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  8. 2022How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysiscited by 42
  9. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  10. 2020Self-Explaining Structures Improve NLP Modelscited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models
  2. 2022Can Pretrained Language Models (Yet) Reason Deductively?
  3. 2024The Role of Deductive and Inductive Reasoning in Large Language Models
  4. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models
  5. 2025Deductive Logic in Language Models: Horizontal vs Vertical Reasoning
  6. 2025Reasoning with Latent Thoughts: On the Power of Looped Transformers
  7. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
  8. 2023Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models