mlsanityData to 7 Oct 2026

Position: Categorical Deep Learning is an Algebraic Theory of All Architectures

arXiv 2402.15332 ↗First posted 23 Feb 2024cs.LG cs.AI math.CTcited by 32

Not on the map. Why?

Abstract (arXiv)

We present our position on the elusive quest for a general-purpose framework for specifying and studying deep learning architectures. Our opinion is that the key attempts made so far lack a coherent bridge between specifying constraints which models must satisfy and specifying their implementations. Focusing on building a such a bridge, we propose to apply category theory -- precisely, the universal algebra of monads valued in a 2-category of parametric maps -- as a single theory elegantly subsuming both of these flavours of neural network design. To defend our position, we show how this theory recovers constraints induced by geometric deep learning, as well as implementations of many architectures drawn from the diverse landscape of neural networks, such as RNNs. We also illustrate how the theory naturally encodes many standard constructs in computer science and automata theory.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

39 of 68 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  4. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2016Group Equivariant Convolutional Networkscited by 1,154
  7. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  8. 2015Automatic differentiation in machine learning: a surveycited by 846
  9. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  10. 2018Spherical CNNscited by 452
  11. 2018Invariant and Equivariant Graph Networkscited by 339
  12. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
27 more
  1. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 223
  2. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  3. 2019What Can Neural Networks Reason About?cited by 155
  4. 2019Deep Scale-spaces: Equivariance Over Scalecited by 113
  5. 2020Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphscited by 90
  6. 2023Everything is Connected: Graph Neural Networkscited by 85
  7. 2022Graph Neural Networks are Dynamic Programmerscited by 55
  8. 2017Backprop as Functor: A compositional perspective on supervised learningcited by 44
  9. 2021Categorical Foundations of Gradient-Based Learningcited by 44
  10. 2023Architectures of Topological Deep Learning: A Survey of Message-Passing Topological Neural Networkscited by 43
  11. 2018On Characterizing the Capacity of Neural Networks using Algebraic Topologycited by 37
  12. 2020Natural Graph Networkscited by 35
  13. 2017Disintegration and Bayesian Inversion via String Diagramscited by 32
  14. 2021Noether Networks: Meta-Learning Useful Conserved Quantitiescited by 26
  15. 2018Causal Inference by String Diagram Surgerycited by 23
  16. 2018The simple essence of automatic differentiationcited by 22
  17. 2017A Convenient Category for Higher-Order Probability Theorycited by 14
  18. 2023Asynchronous Algorithmic Alignment with Cocyclescited by 13
  19. 2022Towards a Grounded Theory of Causation for Embodied AIcited by 9
  20. 2023Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networkscited by 5
  21. 2019Compositionality for Recursive Neural Networkscited by 4
  22. 2023The Compositional Structure of Bayesian Inferencecited by 3
  23. 2021Functorial String Diagrams for Reverse-Mode Automatic Differentiationcited by 2
  24. 2022Space-time tradeoffs of lenses and optics via higher category theorycited by 2
  25. 2023Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networkscited by 2
  26. 2021CHAD for Expressive Total Languagescited by 1
  27. 2021Balancing Fairness and Robustness via Partial Invariancecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 20.

32 papers.

Sort:

By area:

  1. 2024Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structurescited by 20
  2. 2024Towards Compositional Interpretability for XAIcited by 10
  3. 2025Large Causal Models from Large Language Modelscited by 8
  4. 2026Identifiable Equivariant Networks are Layerwise Equivariantcited by 7
  5. 2026Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligencecited by 3
  6. 2024Token Space: A Category Theory Framework for AI Computationscited by 2
  7. 2025Topos Theory for Generative AI and LLMscited by 2
  8. 2024The Topos of Transformer Networkscited by 1
  9. 2025Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networkscited by 1
  10. 2026Decidable By Construction: Design-Time Verification for Truly Fearless Systemscited by 1
  11. 2025The Gauss-Markov Adjunction Provides Categorical Semantics of Residuals in Supervised Learningcited by 0
  12. 2025Relational inductive biases on attention mechanismscited by 0
20 more
  1. 2025A document is worth a structured record: Principled inductive bias design for document recognitioncited by 0
  2. 2025Higher Gauge Flow Modelscited by 0
  3. 2025Graph Lineages and Skeletal Graph Productscited by 0
  4. 2026CSQL: Mapping Documents into Causal Databasescited by 0
  5. 2026Architecture Before the Formula: Individuating Neural Architecture Beyond the Composite Mapcited by 0
  6. 2026Modeling Topological Impact on Node Attribute Distributions in Attributed Graphscited by 0
  7. 2026Product Interaction: An Algebraic Formalism for Deep Learning Architecturescited by 0
  8. 2026Coalgebras for categorical deep learning: Representability and universal approximationcited by 0
  9. 2026Functorial Neural Architectures from Higher Inductive Typescited by 0
  10. 2026Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learningcited by 0
  11. 2026Gaussian Sheaf Neural Networkscited by 0
  12. 2026What are the Right Symmetries for Formal Theorem Proving?cited by 0
  13. 2026The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modelingcited by 0
  14. 2026Operadic consistency: a label-free signal for compositional reasoning failures in LLMscited by 0
  15. 2026A homotopy-type-theoretic generalization of neurosymbolic inferencecited by 0
  16. 2026NeSyCat Torch: A Differentiable Tensor Implementation of Categorical Semantics for Neurosymbolic Learningcited by 0
  17. 2026Graphical Design of Interpretable Architecturescited by 0
  18. 2026Toward Auto-Research: Mining Falsifiable Research Ideas from Paper Knowledge Graphs with Categorical Structurecited by 0
  19. 2026When the Canonical Completion Is Wrong: Formalizing and Measuring the Jump in Large Language Modelscited by 0
  20. 2026Higher Structures in Deep Learningcited by 0