mlsanityData to 14 Oct 2026

Learning Transformer Programs

arXiv 2306.01128 ↗First posted 1 Jun 2023cs.LG cs.CLcited by 41

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Recent research in mechanistic interpretability has attempted to reverse-engineer Transformer models by carefully inspecting network weights and activations. However, these approaches require considerable manual effort and still fall short of providing complete, faithful descriptions of the underlying algorithms. In this work, we introduce a procedure for training Transformers that are mechanistically interpretable by design. We build on RASP [Weiss et al., 2021], a programming language that can be compiled into Transformer weights. Instead of compiling human-written programs into Transformers, we design a modified Transformer that can be trained using gradient-based optimization and then automatically converted into a discrete, human-readable program. We refer to these models as Transformer Programs. To validate our approach, we learn Transformer Programs for a variety of problems, including an in-context learning task, a suite of algorithmic problems (e.g. sorting, recognizing Dyck languages), and NLP tasks including named entity recognition and text classification. The Transformer Programs can automatically find reasonable solutions, performing on par with standard Transformers of comparable size; and, more importantly, they are easy to interpret. To demonstrate these advantages, we convert Transformers into Python programs and use off-the-shelf code analysis tools to debug model errors and identify the "circuits" used to solve different sub-problems. We hope that Transformer Programs open a new path toward the goal of intrinsically interpretable machine learning.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 43 of its 62 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

43 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Progress measures for grokking via mechanistic interpretability12 Jan
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2016Layer Normalizationcited by 5,669
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
29 more
  1. 2022In-context Learning and Induction Headscited by 964
  2. 2019Attention is not Explanationcited by 929
  3. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  4. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  7. 2015Neural Module Networkscited by 376
  8. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 344
  9. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  10. 2023Looped Transformers as Programmable Computerscited by 237
  11. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  12. 2021Thinking Like Transformerscited by 184
  13. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  14. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  15. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  16. 2015Neural Programmer-Interpreterscited by 101
  17. 2021An Interpretability Illusion for BERTcited by 97
  18. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 93
  19. 2020Inductive logic programming at 30: a new introductioncited by 85
  20. 2022Deep Differentiable Logic Gate Networkscited by 67
  21. 2017Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamplescited by 66
  22. 2014Falling Rule Listscited by 59
  23. 2021Sparse Attention with Linear Unitscited by 41
  24. 2021Scalable Rule-Based Representation Learning for Interpretable Classificationcited by 40
  25. 2017Making Neural Programming Architectures Generalize via Recursioncited by 35
  26. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  27. 2017An Empirical Evaluation of Rule Extraction from Recurrent Neural Networkscited by 21
  28. 2019Learning Algorithms via Neural Logic Networkscited by 17
  29. 2021Neurosymbolic Transformers for Multi-Agent Communicationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 24; 2025: 5; 2026: 8.

41 papers.

Sort:

By area:

  1. 2023Large Language Modelscited by 780
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  3. 2023What Formal Languages Can Transformers Express? A Surveycited by 124
  4. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  5. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  6. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  7. 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 49
  8. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 48
  9. 2024Simulation of Graph Algorithms with Looped Transformerscited by 30
  10. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  11. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  12. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 23
29 more
  1. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  2. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  3. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  4. 2024Simulating Hard Attention Using Soft Attentioncited by 15
  5. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 14
  6. 2024Discrete Neural Algorithmic Reasoningcited by 9
  7. 2024KV Shifting Attention Enhances Language Modelingcited by 8
  8. 2025Selective Induction Heads: How Transformers Select Causal Structures In Contextcited by 8
  9. 2024Integration of Mamba and Transformer -- MAT for Long-Short Range Time Series Forecasting with Application to Weather Dynamicscited by 7
  10. 2024ALTA: Compiler-Based Analysis of Transformerscited by 7
  11. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
  12. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  13. 2024Understanding the Logic of Direct Preference Alignment through Logiccited by 5
  14. 2025On the Emergence of Induction Heads for In-Context Learningcited by 5
  15. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  16. 2025Gradient-Based Program Repair: Fixing Bugs in Continuous Program Spacescited by 4
  17. 2024Algorithmic Language Models with Neurally Compiled Librariescited by 2
  18. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  19. 2026Invariant Learning Dynamics of Transformers in Inductive Reasoning Taskscited by 2
  20. 2024Representing Rule-based Chatbots with Transformerscited by 1
  21. 2026Transformers Learn Latent Mixture Models In-Context via Mirror Descentcited by 1
  22. 2026Explaining Attention with Program Synthesiscited by 1
  23. 2026Induction Heads Interpolate N-Gramscited by 1
  24. 2024A mechanistically interpretable neural network for regulatory genomicscited by 0
  25. 2024Rethinking Associative Memory Mechanism in Induction Headcited by 0
  26. 2026Synthesis and Verification of Transformer Programs (Technical Report)cited by 0
  27. 2026How Transformers Learn to Plan via Multi-Token Predictioncited by 0
  28. 2026Can Graph Learning Learn Circuits?cited by 0
  29. 2026Latest Exact Match Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Discovering Interpretable Algorithms by Decompiling Transformers to RASP
  2. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models
  3. 2024Do Efficient Transformers Really Save Computation?
  4. 2024Mechanistic Interpretability of Binary and Ternary Transformers
  5. 2023Can Transformers Learn to Solve Problems Recursively?
  6. 2026Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability
  7. 2026Individual Parameters in Weight-Sparse Transformers Appear Interpretable
  8. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning