mlsanityData to 7 Oct 2026

Tracr: Compiled Transformers as a Laboratory for Interpretability

arXiv 2301.05062 ↗First posted 12 Jan 2023cs.LG cs.AI stat.MLcited by 99

Not on the map. Why?

Abstract (arXiv)

We show how to "compile" human-readable programs into standard decoder-only transformer models. Our compiler, Tracr, generates models with known structure. This structure can be used to design experiments. For example, we use it to study "superposition" in transformers that execute multi-step algorithms. Additionally, the known structure of Tracr-compiled models can serve as ground-truth for evaluating interpretability methods. Commonly, because the "programs" learned by transformers are unknown it is unclear whether an interpretation succeeded. We demonstrate our approach by implementing and examining programs including computing token frequencies, sorting, and parenthesis checking. We provide an open-source implementation of Tracr at https://github.com/google-deepmind/tracr.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 12 Jan 2023.

Builds on

26 of 52 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Layer Normalizationcited by 5,579
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2022Toy Models of Superpositioncited by 928
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  11. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
14 more
  1. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  2. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  3. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  4. 2021Thinking Like Transformerscited by 182
  5. 2020Debugging Tests for Model Explanationscited by 146
  6. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  7. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  8. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  9. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  10. 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
  11. 2022Polysemanticity and Capacity in Neural Networkscited by 74
  12. 2020Towards falsifiable interpretability researchcited by 53
  13. 2021Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networkscited by 48
  14. 2019Evaluating Explanation Without Ground Truth in Interpretable Machine Learningcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 42; 2025: 18; 2026: 21.

99 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  4. 2023Looped Transformers as Programmable Computerscited by 224
  5. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  6. 2023Localizing Model Behavior with Path Patchingcited by 182
  7. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  8. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  9. 2023What Formal Languages Can Transformers Express? A Surveycited by 120
  10. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  11. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  12. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
38 more
  1. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  2. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 65
  3. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  4. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  5. 2023From task structures to world models: What do LLMs know?cited by 46
  6. 2023Learning Transformer Programscited by 40
  7. 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
  8. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  9. 2024Simulation of Graph Algorithms with Looped Transformerscited by 29
  10. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  11. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  12. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  13. 2024Universal Length Generalization with Turing Programscited by 25
  14. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  15. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decompositioncited by 23
  16. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
  17. 2023Red Teaming Deep Neural Networks with Feature Synthesis Toolscited by 21
  18. 2024Opening the AI black box: program synthesis via mechanistic interpretabilitycited by 19
  19. 2024What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Taskscited by 19
  20. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  21. 2024Transformer Circuit Faithfulness Metrics are not Robustcited by 18
  22. 2023Trainable Transformer in Transformercited by 17
  23. 2024Can Transformers Reason Logically? A Study in SAT Solvingcited by 16
  24. 2024Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Modelscited by 13
  25. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
  26. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  27. 2024Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularitycited by 11
  28. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelscited by 9
  29. 2024Transformers Meet Relational Databasescited by 9
  30. 2024Discrete Neural Algorithmic Reasoningcited by 8
  31. 2024Transformers as Transducerscited by 8
  32. 2026All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMscited by 8
  33. 20233VL: Using Trees to Improve Vision-Language Models' Interpretabilitycited by 7
  34. 2024ALTA: Compiler-Based Analysis of Transformerscited by 7
  35. 2025Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMscited by 6
  36. 2025Whither symbols in the era of advanced neural networks?cited by 6
  37. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  38. 2025Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial Vulnerabilitycited by 6

50 most cited of 99 All 99 (JSON)