# Categorical Foundations of Gradient-Based Learning

arXiv: https://arxiv.org/abs/2103.01931 · HTML: https://mlsanity.com/paper/2103.01931/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 74 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Backprop as Functor: A compositional perspective on supervised learning](https://mlsanity.com/paper/1711.10455/) (2017)
- [Towards Verified Artificial Intelligence](https://mlsanity.com/paper/1606.08514/) (2016)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [Reverse derivative categories](https://mlsanity.com/paper/1910.07065/) (2019)
- [Compositional game theory](https://mlsanity.com/paper/1603.04641/) (2016)
- [Lenses and Learners](https://mlsanity.com/paper/1903.03671/) (2019)
- [Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits](https://mlsanity.com/paper/2101.10488/) (2021)
- [Compositional Deep Learning](https://mlsanity.com/paper/1907.08292/) (2019)
- [Profunctor Optics, a Categorical Update](https://mlsanity.com/paper/2001.07488/) (2020)
- [Categorical Stochastic Processes and Likelihood](https://mlsanity.com/paper/2005.04735/) (2020)
- [Differentiable Causal Computations via Delayed Trace](https://mlsanity.com/paper/1903.01093/) (2019)
- [Limits of bimorphic lenses](https://mlsanity.com/paper/1808.05545/) (2018)
- [Diagrammatic Semantics for Digital Circuits](https://mlsanity.com/paper/1703.10247/) (2017)

## Built on by (most cited first, 30 of 44)

- [Category Theory in Machine Learning](https://mlsanity.com/paper/2106.07032/) (2021)
- [Position: Categorical Deep Learning is an Algebraic Theory of All Architectures](https://mlsanity.com/paper/2402.15332/) (2024)
- [Position: LLMs Must Use Functor-Based and RAG-Driven Bias Mitigation for Fairness](https://mlsanity.com/paper/2603.07368/) (2026)
- [Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures](https://mlsanity.com/paper/2402.05424/) (2024)
- [Categorical Foundations of Explainable AI: A Unifying Theory](https://mlsanity.com/paper/2304.14094/) (2023)
- [FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness](https://mlsanity.com/paper/2412.03317/) (2024)
- [On Meta-Prompting](https://mlsanity.com/paper/2312.06562/) (2023)
- [Diagrammatic Differentiation for Quantum Machine Learning](https://mlsanity.com/paper/2103.07960/) (2021)
- [Categories of Differentiable Polynomial Circuits for Machine Learning](https://mlsanity.com/paper/2203.06430/) (2022)
- [Pooling Image Datasets With Multiple Covariate Shift and Imbalance](https://mlsanity.com/paper/2403.02598/) (2024)
- [Reinforcement Learning in Categorical Cybernetics](https://mlsanity.com/paper/2404.02688/) (2024)
- [A Pattern Language for Machine Learning Tasks](https://mlsanity.com/paper/2407.02424/) (2024)
- [The Compositional Structure of Bayesian Inference](https://mlsanity.com/paper/2305.06112/) (2023)
- [Deep Learning with Parametric Lenses](https://mlsanity.com/paper/2404.00408/) (2024)
- [On the Anatomy of Attention](https://mlsanity.com/paper/2407.02423/) (2024)
- [Compositional Active Inference II: Polynomial Dynamics. Approximate Inference Doctrines](https://mlsanity.com/paper/2208.12173/) (2022)
- [Space-time tradeoffs of lenses and optics via higher category theory](https://mlsanity.com/paper/2209.09351/) (2022)
- [Enriching Disentanglement: From Logical Definitions to Quantitative Metrics](https://mlsanity.com/paper/2305.11512/) (2023)
- [Token Space: A Category Theory Framework for AI Computations](https://mlsanity.com/paper/2404.11624/) (2024)
- [Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learning](https://mlsanity.com/paper/2408.13376/) (2024)
- [Generalized Optimization: A First Step Towards Category Theoretic Learning Theory](https://mlsanity.com/paper/2109.10262/) (2021)
- [Deep neural networks as nested dynamical systems](https://mlsanity.com/paper/2111.01297/) (2021)
- [A category theory framework for Bayesian learning](https://mlsanity.com/paper/2111.14293/) (2021)
- [The Topos of Transformer Networks](https://mlsanity.com/paper/2403.18415/) (2024)
- [Logic Explanation of AI Classifiers by Categorical Explaining Functors](https://mlsanity.com/paper/2503.16203/) (2025)
- [Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks](https://mlsanity.com/paper/2505.09326/) (2025)
- [A Probabilistic Generative Model of Free Categories](https://mlsanity.com/paper/2205.04545/) (2022)
- [Graph Convolutional Neural Networks as Parametric CoKleisli morphisms](https://mlsanity.com/paper/2212.00542/) (2022)
- [Algebraic Dynamical Systems in Machine Learning](https://mlsanity.com/paper/2311.03118/) (2023)
- [Generalized Gradient Descent is a Hypergraph Functor](https://mlsanity.com/paper/2403.19845/) (2024)

## Abstract (arXiv)

We propose a categorical semantics of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum, as well as a variety of loss functions such as as MSE and Softmax cross-entropy, shedding new light on their similarities and differences. Our approach to gradient-based learning has examples generalising beyond the familiar continuous domains (modelled in categories of smooth maps) and can be realized in the discrete setting of boolean circuits. Finally, we demonstrate the practical significance of our framework with an implementation in Python.