mlsanityData to 7 Oct 2026

The Scaling Properties of Implicit Deductive Reasoning in Transformers

arXiv 2605.04330 ↗First posted 5 May 2026cs.AI cs.CC cs.LOcited by 1

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

We investigate the scaling properties of implicit deductive reasoning over Horn clauses in depth-bounded Transformers. By discouraging the reliance on statistical shortcuts via counterfactual data augmentation, and promoting the learning of shared reasoning primitives across direct and CoT modes, we find that in sufficiently deep models with a bidirectional prefix mask, implicit reasoning approaches explicit CoT performance across graph topologies and problem widths, though CoT remains necessary for depth extrapolation. These findings represent a step toward achieving better compositional reasoning in Transformers. The code and models to reproduce this work are available at: https://github.com/envomp/Implicit-Deductive-Reasoning-in-Transformers

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 97 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

62 of 97 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Effective Reasoning Chains Reduce Intrinsic Dimensionality9 Feb
  2. 2024Transformers Provably Solve Parity Efficiently with Chain of Thought11 Oct
  3. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning18 Sep
  4. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problems20 Feb
  5. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective24 May
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
38 more
  1. 2022Toy Models of Superpositioncited by 928
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  4. 2018Universal Transformerscited by 613
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  6. 2020Hopfield Networks is All You Needcited by 461
  7. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  8. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  9. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  10. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  11. 2022Transformers Learn Shortcuts to Automatacited by 260
  12. 2023Looped Transformers as Programmable Computerscited by 224
  13. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  14. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  15. 2024The Illusion of State in State-Space Modelscited by 181
  16. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  17. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 161
  18. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  19. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  20. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  21. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  22. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  23. 2022Faithful Reasoning Using Large Language Modelscited by 130
  24. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  25. 2023How do Language Models Bind Entities in Context?cited by 102
  26. 2022On the Paradox of Learning to Reason from Datacited by 93
  27. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 71
  28. 2023Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcutscited by 63
  29. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  30. 2025Magistralcited by 57
  31. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  32. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answeringcited by 38
  33. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  34. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  35. 2023On the Expressivity Role of LayerNorm in Transformers' Attentioncited by 32
  36. 2024Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Modelscited by 25
  37. 2024Your Transformer is Secretly Linearcited by 23
  38. 2025Grokking in the Wild: Data Augmentation for Real-World Multi-Hop Reasoning with Transformerscited by 12

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026High-probability guarantees for linear accessibility in feature superpositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
  2. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithms
  3. 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
  4. 2024Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts
  5. 2023Examining the Emergence of Deductive Reasoning in Generative Language Models
  6. 2025Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers
  7. 2026Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers
  8. 2025Uncovering Graph Reasoning in Decoder-only Transformers with Circuit Tracing