# Higher Order Linear Transformer

arXiv: https://arxiv.org/abs/2010.14816 · HTML: https://mlsanity.com/paper/2010.14816/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 1 of 1)

- [Linearizing Large Language Models](https://mlsanity.com/paper/2405.06640/) (2024)

## Abstract (arXiv)

Following up on the linear transformer part of the article from Katharopoulos et al., that takes this idea from Shen et al., the trick that produces a linear complexity for the attention mechanism is re-used and extended to a second-order approximation of the softmax normalization.