mlsanityData to 7 Oct 2026

Categorical Foundations of Gradient-Based Learning

arXiv 2103.01931 ↗First posted 2 Mar 2021cs.LG math.CTcited by 44

Not on the map. Why?

Abstract (arXiv)

We propose a categorical semantics of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach to gradient-based learning has examples generalising beyond the familiar continuous domains (modelled in categories of smooth maps) and can be realized in the discrete setting of boolean circuits. Finally, we demonstrate the practical significance of our framework with an implementation in Python.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 19 of its 74 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

19 of 74 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  3. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  4. 2016Learning to learn by gradient descent by gradient descentcited by 616
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  6. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  7. 2017Backprop as Functor: A compositional perspective on supervised learningcited by 44
  8. 2016Towards Verified Artificial Intelligencecited by 36
  9. 2018The simple essence of automatic differentiationcited by 22
  10. 2019Reverse derivative categoriescited by 15
  11. 2016Compositional game theorycited by 13
  12. 2019Lenses and Learnerscited by 11
7 more
  1. 2021Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuitscited by 10
  2. 2019Compositional Deep Learningcited by 7
  3. 2020Profunctor Optics, a Categorical Updatecited by 6
  4. 2020Categorical Stochastic Processes and Likelihoodcited by 6
  5. 2019Differentiable Causal Computations via Delayed Tracecited by 5
  6. 2018Limits of bimorphic lensescited by 4
  7. 2017Diagrammatic Semantics for Digital Circuitscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 5; 2024: 12; 2025: 9; 2026: 8.

44 papers.

Sort:

By area:

  1. 2021Category Theory in Machine Learningcited by 36
  2. 2024Position: Categorical Deep Learning is an Algebraic Theory of All Architecturescited by 32
  3. 2024Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architecturescited by 8
  4. 2026Position: LLMs Must Use Functor-Based and RAG-Driven Bias Mitigation for Fairnesscited by 8
  5. 2023Categorical Foundations of Explainable AI: A Unifying Theorycited by 7
  6. 2024FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awarenesscited by 7
  7. 2023On Meta-Promptingcited by 6
  8. 2021Diagrammatic Differentiation for Quantum Machine Learningcited by 5
  9. 2022Categories of Differentiable Polynomial Circuits for Machine Learningcited by 4
  10. 2024Reinforcement Learning in Categorical Cyberneticscited by 4
  11. 2024A Pattern Language for Machine Learning Taskscited by 4
  12. 2023The Compositional Structure of Bayesian Inferencecited by 3
32 more
  1. 2024Pooling Image Datasets With Multiple Covariate Shift and Imbalancecited by 3
  2. 2024Deep Learning with Parametric Lensescited by 3
  3. 2024On the Anatomy of Attentioncited by 3
  4. 2022Compositional Active Inference II: Polynomial Dynamics. Approximate Inference Doctrinescited by 2
  5. 2022Space-time tradeoffs of lenses and optics via higher category theorycited by 2
  6. 2023Enriching Disentanglement: From Logical Definitions to Quantitative Metricscited by 2
  7. 2024Token Space: A Category Theory Framework for AI Computationscited by 2
  8. 2024Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learningcited by 2
  9. 2021Generalized Optimization: A First Step Towards Category Theoretic Learning Theorycited by 1
  10. 2021Deep neural networks as nested dynamical systemscited by 1
  11. 2021A category theory framework for Bayesian learningcited by 1
  12. 2024The Topos of Transformer Networkscited by 1
  13. 2025Logic Explanation of AI Classifiers by Categorical Explaining Functorscited by 1
  14. 2025Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networkscited by 1
  15. 2022A Probabilistic Generative Model of Free Categoriescited by 0
  16. 2022Graph Convolutional Neural Networks as Parametric CoKleisli morphismscited by 0
  17. 2023Algebraic Dynamical Systems in Machine Learningcited by 0
  18. 2024Generalized Gradient Descent is a Hypergraph Functorcited by 0
  19. 2025Developing a Foundation of Vector Symbolic Architectures Using Category Theorycited by 0
  20. 2025The Gauss-Markov Adjunction Provides Categorical Semantics of Residuals in Supervised Learningcited by 0
  21. 2025Higher Gauge Flow Modelscited by 0
  22. 2025Graph Lineages and Skeletal Graph Productscited by 0
  23. 2025Categorical Construction of Logically Verifiable Neural Architecturescited by 0
  24. 2025Reverse Fa`a di Bruno's Formula for Cartesian Reverse Differential Categoriescited by 0
  25. 2025Disentangling Hyperedges through the Lens of Category Theorycited by 0
  26. 2026Modeling Topological Impact on Node Attribute Distributions in Attributed Graphscited by 0
  27. 2026Coalgebras for categorical deep learning: Representability and universal approximationcited by 0
  28. 2026From Specification to Architecture: A Theory Compiler for Knowledge-Guided Machine Learningcited by 0
  29. 2026Functorial Neural Architectures from Higher Inductive Typescited by 0
  30. 2026Ternary Gamma Semirings: From Neural Implementation to Categorical Foundationscited by 0
  31. 2026Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learningcited by 0
  32. 2026Graphical Design of Interpretable Architecturescited by 0