mlsanityData to 7 Oct 2026

On the Anatomy of Attention

arXiv 2407.02423 ↗First posted 2 Jul 2024cs.LG math.CTcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce a category-theoretic diagrammatic formalism in order to systematically relate and reason about machine learning models. Our diagrams present architectures intuitively but without loss of essential detail, where natural relationships between models are captured by graphical transformations, and important differences and similarities can be identified at a glance. In this paper, we focus on attention mechanisms: translating folklore into mathematical derivations, and constructing a taxonomy of attention variants in the literature. As a first example of an empirical investigation underpinned by our formalism, we identify recurring anatomical components of attention, which we exhaustively recombine to explore a space of variations on the attention mechanism.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

34 of 68 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
22 more
  1. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  2. 2021A Survey of Transformerscited by 399
  3. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  4. 2022The Devil in Linear Transformercited by 98
  5. 2019Star-Transformercited by 94
  6. 2021Categorical Foundations of Gradient-Based Learningcited by 44
  7. 2018Modeling Localness for Self-Attention Networkscited by 43
  8. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  9. 2018Causal Inference by String Diagram Surgerycited by 23
  10. 2023Causal models in string diagramscited by 14
  11. 2020A three layer neural network can represent any multivariate functioncited by 10
  12. 2024Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architecturescited by 8
  13. 2023Distilling Text into Circuitscited by 7
  14. 2024An introduction to graphical tensor notation for mechanistic interpretabilitycited by 7
  15. 2018Hypergraph Categoriescited by 6
  16. 2014Interacting Hopf Algebrascited by 5
  17. 2021De Finetti's Theorem in Categorical Probabilitycited by 4
  18. 2020Open Diagrams via Coend Calculuscited by 3
  19. 2015String diagrams for game theorycited by 2
  20. 2018Open Petri Netscited by 2
  21. 2021String Diagrammatic Electrical Circuit Theorycited by 2
  22. 2022Deconstructing the Calculus of Relations with Tape Diagramscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Towards a Categorical Foundation of Deep Learning: A Surveycited by 2
  2. 2026From Mechanistic to Compositional Interpretabilitycited by 1
  3. 2024A Diagrammatic Approach to Improve Computational Efficiency in Group Equivariant Neural Networkscited by 0