mlsanityData to 7 Oct 2026

Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models

arXiv 2310.14491 ↗First posted 23 Oct 2023cs.CLcited by 50

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent work has shown that language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. However, it is unclear whether LMs perform these tasks by cheating with answers memorized from pretraining corpus, or, via a multi-step reasoning mechanism. In this paper, we try to answer this question by exploring a mechanistic interpretation of LMs for multi-step reasoning tasks. Concretely, we hypothesize that the LM implicitly embeds a reasoning tree resembling the correct reasoning process within it. We test this hypothesis by introducing a new probing approach (called MechanisticProbe) that recovers the reasoning tree from the model's attention patterns. We use our probe to analyze two LMs: GPT-2 on a synthetic task (k-th smallest element), and LLaMA on two simple language-based reasoning tasks (ProofWriter & AI2 Reasoning Challenge). We show that MechanisticProbe is able to detect the information of the reasoning tree from the model's attentions for most examples, suggesting that the LM indeed is going through a process of multi-step reasoning within its architecture in many cases.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 30 of its 50 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

30 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement12 Oct
  2. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May
  3. 2023RECKONING: Reasoning through Dynamic Knowledge Encoding10 May
  4. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  5. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  8. 2020Quantifying Attention Flow in Transformerscited by 759
  9. 2019Designing and Interpreting Probes with Control Taskscited by 608
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  11. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  12. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
13 more
  1. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  2. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  3. 2019On Identifiability in Transformerscited by 115
  4. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  5. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  6. 2022Probing for the Usage of Grammatical Numbercited by 57
  7. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  8. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  9. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  10. 2021Attention Flows are Shapley Value Explanationscited by 30
  11. 2022Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?cited by 20
  12. 2021On-the-Fly Attention Modulation for Neural Generationcited by 12
  13. 2021Bird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approachcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 21; 2026: 10.

50 papers.

Sort:

By area:

  1. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  2. 2024How do Large Language Models Handle Multilingualism?cited by 148
  3. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  4. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  5. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  6. 2025A Survey on Latent Reasoningcited by 71
  7. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  8. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  9. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 42
  10. 2024Locate-then-edit for Multi-hop Factual Recall under Knowledge Editingcited by 36
  11. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  12. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
38 more
  1. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  2. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  3. 2025Base Models Know How to Reason, Thinking Models Learn Whencited by 25
  4. 2024Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregationcited by 22
  5. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  6. 2024Transformers Struggle to Learn to Searchcited by 19
  7. 2025Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamicscited by 16
  8. 2024Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neuronscited by 15
  9. 2024AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimationcited by 14
  10. 2025General agents contain world modelscited by 14
  11. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  12. 2024Abrupt Learning in Transformers: A Case Study on Matrix Completioncited by 11
  13. 2025Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoningcited by 11
  14. 2024Evaluating the Ability of Large Language Models to Reason about Cardinal Directionscited by 10
  15. 2026Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Taskscited by 10
  16. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learnerscited by 9
  17. 2025Enhancing Transformers for Generalizable First-Order Logical Entailmentcited by 6
  18. 2025Punctuation and Predicates in Language Modelscited by 5
  19. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Modelscited by 4
  20. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  21. 2026Do LLMs Encode Functional Importance of Reasoning Tokens?cited by 4
  22. 2025Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Trainingcited by 3
  23. 2025Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMscited by 3
  24. 2025ACE: Attribution-Controlled Knowledge Editing for Multi-hop Factual Recallcited by 3
  25. 2026Capabilities and Fundamental Limits of Latent Chain-of-Thoughtcited by 3
  26. 2025CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrationscited by 2
  27. 2025Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainabilitycited by 2
  28. 2025Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2cited by 2
  29. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  30. 2026Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Modelscited by 1
  31. 2026Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalizationcited by 1
  32. 2026When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Modelscited by 1
  33. 2024Identifying Sub-networks in Neural Networks via Functionally Similar Representationscited by 0
  34. 2024Information Anxiety in Large Language Modelscited by 0
  35. 2026Multi-Hop Knowledge Composition is Bound by Pretraining Exposurecited by 0
  36. 2026Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Modelcited by 0
  37. 2026Beyond Fixed Directions: Adaptive Representation Analysis of Reasoning and Memorization in LLMscited by 0
  38. 2026RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
  2. 2024First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning
  3. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
  4. 2024textttLM^texttt2: A Simple Society of Language Models Solves Complex Reasoning
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
  6. 2026Towards a Mechanistic Understanding of Propositional Logical Reasoning in Large Language Models
  7. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  8. 2024ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure