mlsanityData to 14 Oct 2026

The Topos of Transformer Networks

arXiv 2403.18415 ↗First posted 27 Mar 2024cs.LG math.CTcited by 1

Not on the map. Why?

Abstract (arXiv)

The transformer neural network has significantly out-shined all other neural network architectures as the engine behind large language models. We provide a theoretical analysis of the expressivity of the transformer architecture through the lens of topos theory. From this viewpoint, we show that many common neural network architectures, such as the convolutional, recurrent and graph convolutional networks, can be embedded in a pretopos of piecewise-linear functions, but that the transformer necessarily lives in its topos completion. In particular, this suggests that the two network families instantiate different fragments of logic: the former are first order, whereas transformers are higher-order reasoners. Furthermore, we draw parallels with architecture search and gradient descent, integrating our analysis in the framework of cybernetic agents.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

30 of 51 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2014On the Number of Linear Regions of Deep Neural Networkscited by 381
  8. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 257
  9. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  10. 2022Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNscited by 214
  11. 2022Sheaf Neural Networks with Connection Laplacianscited by 65
  12. 2021Towards Lower Bounds on the Depth of ReLU Neural Networkscited by 56
18 more
  1. 2022Graph Neural Networks are Dynamic Programmerscited by 55
  2. 2021Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networkscited by 48
  3. 2017Backprop as Functor: A compositional perspective on supervised learningcited by 44
  4. 2021Categorical Foundations of Gradient-Based Learningcited by 44
  5. 2020Natural Graph Networkscited by 35
  6. 2020Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplificationcited by 32
  7. 2024Position: Categorical Deep Learning is an Algebraic Theory of All Architecturescited by 32
  8. 2022Neural Networks are Decision Treescited by 17
  9. 2021Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verificationcited by 13
  10. 2019Luck Matters: Understanding Training Dynamics of Deep ReLU Networkscited by 8
  11. 2023Representing Piecewise Linear Functions by Functions with Small Aritycited by 7
  12. 2021Learners' Languagescited by 6
  13. 2023Any Deep ReLU Network is Shallowcited by 6
  14. 2021Topos and Stacks of Deep Neural Networkscited by 5
  15. 2023Unwrapping All ReLU Networkscited by 4
  16. 2023Towards Rigorous Understanding of Neural Networks via Semantics-preserving Transformationscited by 3
  17. 2021Deep neural networks as nested dynamical systemscited by 1
  18. 2021Graphical Piecewise-Linear Algebracited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025The brain-AI convergence: Predictive and generative world models for general-purpose computationcited by 0