# Towards a Categorical Foundation of Deep Learning: A Survey

arXiv: https://arxiv.org/abs/2410.05353 · HTML: https://mlsanity.com/paper/2410.05353/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 83 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [What Can Neural Networks Reason About?](https://mlsanity.com/paper/1905.13211/) (2019)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Sheaf Neural Networks](https://mlsanity.com/paper/2012.06333/) (2020)
- [Formal Algorithms for Transformers](https://mlsanity.com/paper/2207.09238/) (2022)
- [A Survey of Complex-Valued Neural Networks](https://mlsanity.com/paper/2101.12249/) (2021)
- [Category Theory in Machine Learning](https://mlsanity.com/paper/2106.07032/) (2021)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [Reverse derivative categories](https://mlsanity.com/paper/1910.07065/) (2019)
- [Lenses and Learners](https://mlsanity.com/paper/1903.03671/) (2019)
- [Named Tensor Notation](https://mlsanity.com/paper/2102.13196/) (2021)
- [Reverse Derivative Ascent: A Categorical Approach to Learning Boolean Circuits](https://mlsanity.com/paper/2101.10488/) (2021)
- [Graph Tensor Networks: An Intuitive Framework for Designing Large-Scale Neural Learning Systems on Multiple Domains](https://mlsanity.com/paper/2303.13565/) (2023)
- [An introduction to graphical tensor notation for mechanistic interpretability](https://mlsanity.com/paper/2402.01790/) (2024)
- [Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures](https://mlsanity.com/paper/2402.05424/) (2024)
- [Compositional Deep Learning](https://mlsanity.com/paper/1907.08292/) (2019)
- [Fundamental Components of Deep Learning: A category-theoretic approach](https://mlsanity.com/paper/2403.13001/) (2024)
- [Compositionality for Recursive Neural Networks](https://mlsanity.com/paper/1901.10723/) (2019)
- [Nonlinear Sheaf Diffusion in Graph Neural Networks](https://mlsanity.com/paper/2403.00337/) (2024)
- [On the Anatomy of Attention](https://mlsanity.com/paper/2407.02423/) (2024)
- [Transport of Algebraic Structure to Latent Embeddings](https://mlsanity.com/paper/2405.16763/) (2024)
- [Category Theory Using String Diagrams](https://mlsanity.com/paper/1401.7220/) (2014)

## Built on by (most cited first, 2 of 2)

- [Self-Revising Discovery Systems for Science: A Categorical Framework for Agentic Artificial Intelligence](https://mlsanity.com/paper/2606.01444/) (2026)
- [Higher Gauge Flow Models](https://mlsanity.com/paper/2507.16334/) (2025)

## Abstract (arXiv)

The unprecedented pace of machine learning research has lead to incredible advances, but also poses hard challenges. At present, the field lacks strong theoretical underpinnings, and many important achievements stem from ad hoc design choices which are hard to justify in principle and whose effectiveness often goes unexplained. Research debt is increasing and many papers are found not to be reproducible. This thesis is a survey that covers some recent work attempting to study machine learning categorically. Category theory is a branch of abstract mathematics that has found successful applications in many fields, both inside and outside mathematics. Acting as a lingua franca of mathematics and science, category theory might be able to give a unifying structure to the field of machine learning. This could solve some of the aforementioned problems. In this work, we mainly focus on the application of category theory to deep learning. Namely, we discuss the use of categorical optics to model gradient-based learning, the use of categorical algebras and integral transforms to link classical computer science to neural networks, the use of functors to link different layers of abstraction and preserve structure, and, finally, the use of string diagrams to provide detailed representations of neural network architectures.