# Coinductive guide to inductive transformer heads

arXiv: https://arxiv.org/abs/2302.01834 · HTML: https://mlsanity.com/paper/2302.01834/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 53 references that are in the map.
- First posted on arXiv 3 Feb 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Formal Algorithms for Transformers](https://mlsanity.com/paper/2207.09238/) (2022)
- [CertRL: Formalizing Convergence Proofs for Value and Policy Iteration in Coq](https://mlsanity.com/paper/2009.11403/) (2020)
- [Interacting Hopf Algebras](https://mlsanity.com/paper/1403.7048/) (2014)
- [The Brownian motion in the transformer model](https://mlsanity.com/paper/2107.05264/) (2021)
- [A Categorical Reconstruction of Quantum Theory](https://mlsanity.com/paper/1804.02265/) (2018)
- [In Praise of Sequence (Co-)Algebra and its implementation in Haskell](https://mlsanity.com/paper/1812.05878/) (2018)
- [Folding over Neural Networks](https://mlsanity.com/paper/2207.01090/) (2022)

## Abstract (arXiv)

We argue that all building blocks of transformer models can be expressed with a single concept: combinatorial Hopf algebra. Transformer learning emerges as a result of the subtle interplay between the algebraic and coalgebraic operations of the combinatorial Hopf algebra. Viewed through this lens, the transformer model becomes a linear time-invariant system where the attention mechanism computes a generalized convolution transform and the residual stream serves as a unit impulse. Attention-only transformers then learn by enforcing an invariant between these two paths. We call this invariant Hopf coherence. Due to this, with a degree of poetic license, one could call combinatorial Hopf algebras "tensors with a built-in loss function gradient". This loss function gradient occurs within the single layers and no backward pass is needed. This is in contrast to automatic differentiation which happens across the whole graph and needs a explicit backward pass. This property is the result of the fact that combinatorial Hopf algebras have the surprising property of calculating eigenvalues by repeated squaring.