# Topos and Stacks of Deep Neural Networks

arXiv: https://arxiv.org/abs/2106.14587 · HTML: https://mlsanity.com/paper/2106.14587/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 119 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)](https://mlsanity.com/paper/1412.6632/) (2014)
- [Learning to Compose Neural Networks for Question Answering](https://mlsanity.com/paper/1601.01705/) (2016)
- [GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering](https://mlsanity.com/paper/1902.09506/) (2019)
- [Relational Deep Reinforcement Learning](https://mlsanity.com/paper/1806.01830/) (2018)
- [Attention over learned object embeddings enables complex visual reasoning](https://mlsanity.com/paper/2012.08508/) (2020)
- [Backprop as Functor: A compositional perspective on supervised learning](https://mlsanity.com/paper/1711.10455/) (2017)
- [N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials](https://mlsanity.com/paper/1803.01588/) (2018)
- [Minimal Gated Unit for Recurrent Neural Networks](https://mlsanity.com/paper/1603.09420/) (2016)
- [Simplified Minimal Gated Unit Variations for Recurrent Neural Networks](https://mlsanity.com/paper/1701.03452/) (2017)
- [Discovering objects and their relations from entangled scene representations](https://mlsanity.com/paper/1702.05068/) (2017)
- [Topological Analysis of Syntactic Structures](https://mlsanity.com/paper/1903.05181/) (2019)

## Built on by (most cited first, 5 of 5)

- [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022)
- [Semantic-Native Communication: A Simplicial Complex Perspective](https://mlsanity.com/paper/2210.16970/) (2022)
- [The Topos of Transformer Networks](https://mlsanity.com/paper/2403.18415/) (2024)
- [Logical Information Cells I](https://mlsanity.com/paper/2108.04751/) (2021)
- [Joint Semantic-Native Communication and Inference via Minimal Simplicial Structures](https://mlsanity.com/paper/2308.16789/) (2023)

## Abstract (arXiv)

Every known artificial deep neural network (DNN) corresponds to an object in a canonical Grothendieck's topos; its learning dynamic corresponds to a flow of morphisms in this topos. Invariance structures in the layers (like CNNs or LSTMs) correspond to Giraud's stacks. This invariance is supposed to be responsible of the generalization property, that is extrapolation from learning data under constraints. The fibers represent pre-semantic categories (Culioli, Thom), over which artificial languages are defined, with internal logics, intuitionist, classical or linear (Girard). Semantic functioning of a network is its ability to express theories in such a language for answering questions in output about input data. Quantities and spaces of semantic information are defined by analogy with the homological interpretation of Shannon's entropy of P.Baudot and D.Bennequin in 2015). They generalize the measures found by Carnap and Bar-Hillel (1952). Amazingly, the above semantical structures are classified by geometric fibrant objects in a closed model category of Quillen, then they give rise to homotopical invariants of DNNs and of their semantic functioning. Intentional type theories (Martin-Loef) organize these objects and fibrations between them. Information contents and exchanges are analyzed by Grothendieck's derivators.