# An introduction to graphical tensor notation for mechanistic interpretability

arXiv: https://arxiv.org/abs/2402.01790 · HTML: https://mlsanity.com/paper/2402.01790/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 39 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 5 of 5)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Graph Tensor Networks: An Intuitive Framework for Designing Large-Scale Neural Learning Systems on Multiple Domains](https://mlsanity.com/paper/2303.13565/) (2023)

## Built on by (most cited first, 8 of 8)

- [On the Anatomy of Attention](https://mlsanity.com/paper/2407.02423/) (2024)
- [Towards a Categorical Foundation of Deep Learning: A Survey](https://mlsanity.com/paper/2410.05353/) (2024)
- [Broadcast Product: Redefining Shape-aligned Element-wise Multiplication and Beyond](https://mlsanity.com/paper/2409.17502/) (2024)
- [Very Basics of Tensors with Graphical Notations: Unfolding, Calculations, and Decompositions](https://mlsanity.com/paper/2411.16094/) (2024)
- [Optimal Invariant Bases for Atomistic Machine Learning](https://mlsanity.com/paper/2503.23515/) (2025)
- [Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability](https://mlsanity.com/paper/2608.30505/) (2026)
- [Graphical einops: bridging tensor networks and computation graphs](https://mlsanity.com/paper/2605.31485/) (2026)
- [Graphical Design of Interpretable Architectures](https://mlsanity.com/paper/2608.18936/) (2026)

## Abstract (arXiv)

Graphical tensor notation is a simple way of denoting linear operations on tensors, originating from physics. Modern deep learning consists almost entirely of operations on or between tensors, so easily understanding tensor operations is quite important for understanding these systems. This is especially true when attempting to reverse-engineer the algorithms learned by a neural network in order to understand its behavior: a field known as mechanistic interpretability. It's often easy to get confused about which operations are happening between tensors and lose sight of the overall structure, but graphical tensor notation makes it easier to parse things at a glance and see interesting equivalences. The first half of this document introduces the notation and applies it to some decompositions (SVD, CP, Tucker, and tensor network decompositions), while the second half applies it to some existing some foundational approaches for mechanistically understanding language models, loosely following ``A Mathematical Framework for Transformer Circuits'', then constructing an example ``induction head'' circuit in graphical tensor notation.