# On the Anatomy of Attention

arXiv: https://arxiv.org/abs/2407.02423 · HTML: https://mlsanity.com/paper/2407.02423/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 68 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [The Devil in Linear Transformer](https://mlsanity.com/paper/2210.10340/) (2022)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)
- [Categorical Foundations of Gradient-Based Learning](https://mlsanity.com/paper/2103.01931/) (2021)
- [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018)
- [Hard-Coded Gaussian Attention for Neural Machine Translation](https://mlsanity.com/paper/2005.00742/) (2020)
- [Causal Inference by String Diagram Surgery](https://mlsanity.com/paper/1811.08338/) (2018)
- [Causal models in string diagrams](https://mlsanity.com/paper/2304.07638/) (2023)
- [A three layer neural network can represent any multivariate function](https://mlsanity.com/paper/2012.03016/) (2020)
- [An introduction to graphical tensor notation for mechanistic interpretability](https://mlsanity.com/paper/2402.01790/) (2024)
- [Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures](https://mlsanity.com/paper/2402.05424/) (2024)
- [Hypergraph Categories](https://mlsanity.com/paper/1806.08304/) (2018)
- [Distilling Text into Circuits](https://mlsanity.com/paper/2301.10595/) (2023)
- [Interacting Hopf Algebras](https://mlsanity.com/paper/1403.7048/) (2014)
- [De Finetti's Theorem in Categorical Probability](https://mlsanity.com/paper/2105.02639/) (2021)
- [Open Diagrams via Coend Calculus](https://mlsanity.com/paper/2004.04526/) (2020)

## Built on by (most cited first, 3 of 3)

- [Towards a Categorical Foundation of Deep Learning: A Survey](https://mlsanity.com/paper/2410.05353/) (2024)
- [From Mechanistic to Compositional Interpretability](https://mlsanity.com/paper/2605.08934/) (2026)
- [A Diagrammatic Approach to Improve Computational Efficiency in Group Equivariant Neural Networks](https://mlsanity.com/paper/2412.10837/) (2024)

## Abstract (arXiv)

We introduce a category-theoretic diagrammatic formalism in order to systematically relate and reason about machine learning models. Our diagrams present architectures intuitively but without loss of essential detail, where natural relationships between models are captured by graphical transformations, and important differences and similarities can be identified at a glance. In this paper, we focus on attention mechanisms: translating folklore into mathematical derivations, and constructing a taxonomy of attention variants in the literature. As a first example of an empirical investigation underpinned by our formalism, we identify recurring anatomical components of attention, which we exhaustively recombine to explore a space of variations on the attention mechanism.