# Position: Categorical Deep Learning is an Algebraic Theory of All Architectures

arXiv: https://arxiv.org/abs/2402.15332 · HTML: https://mlsanity.com/paper/2402.15332/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 68 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Invariant and Equivariant Graph Networks](https://mlsanity.com/paper/1812.09902/) (2018)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [What Can Neural Networks Reason About?](https://mlsanity.com/paper/1905.13211/) (2019)
- [Machine Learning on Graphs: A Model and Comprehensive Taxonomy](https://mlsanity.com/paper/2005.03675/) (2020)
- [Deep Scale-spaces: Equivariance Over Scale](https://mlsanity.com/paper/1905.11697/) (2019)
- [Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs](https://mlsanity.com/paper/2003.05425/) (2020)
- [Everything is Connected: Graph Neural Networks](https://mlsanity.com/paper/2301.08210/) (2023)
- [Graph Neural Networks are Dynamic Programmers](https://mlsanity.com/paper/2203.15544/) (2022)
- [Backprop as Functor: A compositional perspective on supervised learning](https://mlsanity.com/paper/1711.10455/) (2017)
- [Categorical Foundations of Gradient-Based Learning](https://mlsanity.com/paper/2103.01931/) (2021)
- [Architectures of Topological Deep Learning: A Survey of Message-Passing Topological Neural Networks](https://mlsanity.com/paper/2304.10031/) (2023)
- [On Characterizing the Capacity of Neural Networks using Algebraic Topology](https://mlsanity.com/paper/1802.04443/) (2018)
- [Natural Graph Networks](https://mlsanity.com/paper/2007.08349/) (2020)
- [Disintegration and Bayesian Inversion via String Diagrams](https://mlsanity.com/paper/1709.00322/) (2017)
- [Noether Networks: Meta-Learning Useful Conserved Quantities](https://mlsanity.com/paper/2112.03321/) (2021)
- [Causal Inference by String Diagram Surgery](https://mlsanity.com/paper/1811.08338/) (2018)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [A Convenient Category for Higher-Order Probability Theory](https://mlsanity.com/paper/1701.02547/) (2017)
- [Asynchronous Algorithmic Alignment with Cocycles](https://mlsanity.com/paper/2306.15632/) (2023)

## Built on by (most cited first, 30 of 32)

- [Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures](https://mlsanity.com/paper/2407.09468/) (2024)
- [Towards Compositional Interpretability for XAI](https://mlsanity.com/paper/2406.17583/) (2024)
- [Large Causal Models from Large Language Models](https://mlsanity.com/paper/2512.07796/) (2025)
- [Identifiable Equivariant Networks are Layerwise Equivariant](https://mlsanity.com/paper/2601.21645/) (2026)
- [Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligence](https://mlsanity.com/paper/2606.01444/) (2026)
- [Token Space: A Category Theory Framework for AI Computations](https://mlsanity.com/paper/2404.11624/) (2024)
- [Topos Theory for Generative AI and LLMs](https://mlsanity.com/paper/2508.08293/) (2025)
- [The Topos of Transformer Networks](https://mlsanity.com/paper/2403.18415/) (2024)
- [Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks](https://mlsanity.com/paper/2505.09326/) (2025)
- [Decidable By Construction: Design-Time Verification for Truly Fearless Systems](https://mlsanity.com/paper/2603.25414/) (2026)
- [What are the Right Symmetries for Formal Theorem Proving?](https://mlsanity.com/paper/2605.22257/) (2026)
- [The Gauss-Markov Adjunction Provides Categorical Semantics of Residuals in Supervised Learning](https://mlsanity.com/paper/2507.02442/) (2025)
- [Relational inductive biases on attention mechanisms](https://mlsanity.com/paper/2507.04117/) (2025)
- [A document is worth a structured record: Principled inductive bias design for document recognition](https://mlsanity.com/paper/2507.08458/) (2025)
- [Higher Gauge Flow Models](https://mlsanity.com/paper/2507.16334/) (2025)
- [Graph Lineages and Skeletal Graph Products](https://mlsanity.com/paper/2508.00197/) (2025)
- [CSQL: Mapping Documents into Causal Databases](https://mlsanity.com/paper/2601.08109/) (2026)
- [Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers](https://mlsanity.com/paper/2601.11618/) (2026)
- [Modeling Topological Impact on Node Attribute Distributions in Attributed Graphs](https://mlsanity.com/paper/2602.01454/) (2026)
- [Product Interaction: An Algebraic Formalism for Deep Learning Architectures](https://mlsanity.com/paper/2602.02573/) (2026)
- [Coalgebras for categorical deep learning: Representability and universal approximation](https://mlsanity.com/paper/2603.03227/) (2026)
- [Functorial Neural Architectures from Higher Inductive Types](https://mlsanity.com/paper/2603.16123/) (2026)
- [Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning](https://mlsanity.com/paper/2604.07242/) (2026)
- [Gaussian Sheaf Neural Networks](https://mlsanity.com/paper/2605.21435/) (2026)
- [The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling](https://mlsanity.com/paper/2605.28864/) (2026)
- [Operadic consistency: a label-free signal for compositional reasoning failures in LLMs](https://mlsanity.com/paper/2606.13649/) (2026)
- [A homotopy-type-theoretic generalization of neurosymbolic inference](https://mlsanity.com/paper/2606.17851/) (2026)
- [NeSyCat Torch: A Differentiable Tensor Implementation of Categorical Semantics for Neurosymbolic Learning](https://mlsanity.com/paper/2606.19279/) (2026)
- [Graphical Design of Interpretable Architectures](https://mlsanity.com/paper/2608.18936/) (2026)
- [Toward Auto-Research: Mining Falsifiable Research Ideas from Paper Knowledge Graphs with Categorical Structure](https://mlsanity.com/paper/2608.20361/) (2026)

## Abstract (arXiv)

We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures. Our opinion is that the key attempts made so far lack a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building a such a bridge, we propose to apply category theory -- precisely, the universal algebra of monads valued in a 2-category of parametric maps -- as a single theory elegantly subsuming both of these flavours of neural network design. To defend our position, we show how this theory recovers constraints induced by geometric deep learning, as well as implementations of many architectures drawn from the diverse landscape of neural networks, such as RNNs. We also illustrate how the theory naturally encodes many standard constructs in computer science and automata theory.