# Kronecker Attention Networks

arXiv: https://arxiv.org/abs/2007.08442 · HTML: https://mlsanity.com/paper/2007.08442/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 9 of 9)

- [SeaFormer++: Squeeze-enhanced Axial Transformer for Mobile Visual Recognition](https://mlsanity.com/paper/2301.13156/) (2023)
- [SDTP: Semantic-aware Decoupled Transformer Pyramid for Dense Image Prediction](https://mlsanity.com/paper/2109.08963/) (2021)
- [Structured and Fast Optimization: The Kronecker SGD Algorithm](https://mlsanity.com/paper/2305.08001/) (2023)
- [EiGLasso for Scalable Sparse Kronecker-Sum Inverse Covariance Estimation](https://mlsanity.com/paper/2105.09872/) (2021)
- [TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors](https://mlsanity.com/paper/2601.17958/) (2026)
- [Higher-Order Transformers With Kronecker-Structured Attention](https://mlsanity.com/paper/2412.02919/) (2024)
- [Fine-Grained Attention for Weakly Supervised Object Localization](https://mlsanity.com/paper/2104.04952/) (2021)
- [LaViP:Language-Grounded Visual Prompts](https://mlsanity.com/paper/2312.10945/) (2023)
- [Krony-PT: GPT2 compressed with Kronecker Products](https://mlsanity.com/paper/2412.12351/) (2024)

## Abstract (arXiv)

Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by assuming the data follow matrix-variate normal distributions. Based on this new view, we develop Kronecker attention operators (KAOs) that operate on high-order tensor data directly. More importantly, the proposed KAOs lead to dramatic reductions in computational resources. Experimental results show that our methods reduce the amount of required computational resources by a factor of hundreds, with larger factors for higher-dimensional and higher-order data. Results also show that networks with KAOs outperform models without attention, while achieving competitive performance as those with original attention operators.