# Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning

arXiv: https://arxiv.org/abs/2605.27259 · HTML: https://mlsanity.com/paper/2605.27259/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs](https://mlsanity.com/paper/2202.04579/) (2022)
- [Topological Deep Learning: Going Beyond Graph Data](https://mlsanity.com/paper/2206.00606/) (2022)
- [Sheaf Neural Networks with Connection Laplacians](https://mlsanity.com/paper/2206.08702/) (2022)
- [Simplicial Attention Networks](https://mlsanity.com/paper/2204.09455/) (2022)
- [HEGEL: Hypergraph Transformer for Long Document Summarization](https://mlsanity.com/paper/2210.04126/) (2022)
- [Generalized Simplicial Attention Neural Networks](https://mlsanity.com/paper/2309.02138/) (2023)
- [Copresheaf Topological Neural Networks: A Generalized Deep Learning Framework](https://mlsanity.com/paper/2505.21251/) (2025)
- [Large Causal Models from Large Language Models](https://mlsanity.com/paper/2512.07796/) (2025)
- [PROMETHEUS: Automating Deep Causal Research Integrating Text, Data and Models](https://mlsanity.com/paper/2605.12835/) (2026)

## Built on by (most cited first, 3 of 3)

- [Odyssey: Constructing Verifiable Local Truth-Preserving Foundation Models](https://mlsanity.com/paper/2606.27593/) (2026)
- [Learning in Infinitesimal Non-Compositional Sketches](https://mlsanity.com/paper/2607.15107/) (2026)
- [Learning Transfers: Kan Extensions of Topological Invariants of Learned Representations](https://mlsanity.com/paper/2606.07627/) (2026)

## Abstract (arXiv)

We propose Kan Extension Transformers (KETs) as a categorical design language for a diverse group of Transformer implementations. A layer can be viewed generally as a weighted structured extension operator: attention uses token neighborhoods, geometric mixing uses sparse incidences, and KET uses simplicial sources. This operator is an actual enriched left Kan extension only when the source values are functorial, the weights are representable hom-objects (or the specified profunctor action), and aggregation realizes the corresponding coend; otherwise ``Kan-style'' denotes an interpretation rather than an identity theorem. Predict-detach blocks gradients through a predictive carrier and avoids transporting teacher-forced hidden states, but detach alone does not make a noncausal update strictly autoregressive: every carrier consumed at target t must also be measurable from the prefix available at t. We evaluate 12 implementations on Penn Treebank, WikiText-2, and WikiText-103 across strict-causal and self-conditioned regimes, using widths d=64,256 and depths L=2,8,16 across the reported studies. Quadratic KET is strongest among the compared strict-causal architectures on WikiText-2 and WikiText-103; the largest cross-regime gains arise from additional self-conditioning information, not neighborhood design alone.