mlsanityData to 7 Oct 2026

A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task

arXiv 2402.11917 ↗First posted 19 Feb 2024cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Transformers demonstrate impressive performance on a range of reasoning benchmarks. To evaluate the degree to which these abilities are a result of actual reasoning, existing work has focused on developing sophisticated benchmarks for behavioral studies. However, these studies do not provide insights into the internal mechanisms driving the observed capabilities. To improve our understanding of the internal mechanisms of transformers, we present a comprehensive mechanistic analysis of a transformer trained on a synthetic reasoning task. We identify a set of interpretable mechanisms the model uses to solve the task, and validate our findings using correlational and causal evidence. Our results suggest that it implements a depth-bounded recurrent mechanisms that operates in parallel and stores intermediate results in selected token positions. We anticipate that the motifs we identified in our synthetic setting can provide valuable insights into the broader operating principles of transformers and thus provide a basis for understanding more complex models.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

32 of 57 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
20 more
  1. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  2. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  3. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  4. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  5. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  6. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  7. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  8. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  9. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  10. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  11. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  12. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  13. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  14. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  15. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  16. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
  17. 2023Understanding and Controlling a Maze-Solving Policy Networkcited by 19
  18. 2023Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Modelscited by 17
  19. 2023Structured World Representations in Maze-Solving Transformerscited by 16
  20. 2024Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 21; 2026: 14.

49 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  3. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  4. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  5. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  6. 2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 33
  7. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  8. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  9. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  10. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  11. 2024Transformers Struggle to Learn to Searchcited by 19
  12. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
37 more
  1. 2024Planning in a recurrent neural network that plays Sokobancited by 17
  2. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  3. 2024Transformers Use Causal World Models in Maze-Solving Taskscited by 13
  4. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoningcited by 10
  5. 2026Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Taskscited by 10
  6. 2025Mechanistic evaluation of Transformers and state space modelscited by 8
  7. 2024On Logical Extrapolation for Mazes with Recurrent and Implicit Networkscited by 7
  8. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  9. 2025Optimizing Multi-Hop Document Retrieval Through Intermediate Representationscited by 5
  10. 2025Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNNcited by 4
  11. 2025In-Context Algebracited by 4
  12. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  13. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  14. 2025Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Spacecited by 3
  15. 2026Sketch2Simulation: Automating Flowsheet Generation via Multi Agent Large Language Modelscited by 3
  16. 2024Interacting Large Language Model Agents. Interpretable Models and Social Learningcited by 2
  17. 2025Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Modelscited by 2
  18. 2025Signatures of human-like processing in Transformer forward passescited by 2
  19. 2025Steering Prepositional Phrases in Language Models: A Case of with-headed Adjectival and Adverbial Complements in Gemma-2cited by 2
  20. 2025Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attentioncited by 2
  21. 2025Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Modelscited by 1
  22. 2025Deductive Logic in Language Models: Horizontal vs Vertical Reasoningcited by 1
  23. 2025Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformerscited by 1
  24. 2026Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformerscited by 1
  25. 2026Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefillingcited by 1
  26. 2026Trustworthy AI Suffers from Invariance Conflicts and Causality is The Solutioncited by 1
  27. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoningcited by 1
  28. 2025A Neuroscience-Inspired Dual-Process Model of Compositional Generalizationcited by 0
  29. 2025Can Language Models Compose Skills In-Context?cited by 0
  30. 2025Exploring Depth Generalization in Large Language Models for Solving Recursive Logic Taskscited by 0
  31. 2026One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformercited by 0
  32. 2026Mechanisms of Misgeneralization in Physical Sequence Modelingcited by 0
  33. 2026A Close Look At World Model Recovery In Supervised Fine-Tuned LLM Plannerscited by 0
  34. 2026Critical Percolation as a Synthetic Data Model for Interpretabilitycited by 0
  35. 2026AIMO Interpretability Challengecited by 0
  36. 2026Grounding latent algorithm routing in transformer reasoningcited by 0
  37. 2026Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes Itcited by 0