# Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation

arXiv: https://arxiv.org/abs/2007.14902 · HTML: https://mlsanity.com/paper/2007.14902/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 21 of 21)

- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents](https://mlsanity.com/paper/2502.06975/) (2025)
- [Why Softmax Attention Outperforms Linear Attention](https://mlsanity.com/paper/2310.11685/) (2023)
- [Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: A Benchmark Study](https://mlsanity.com/paper/2402.19052/) (2024)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [FsaNet: Frequency Self-attention for Semantic Segmentation](https://mlsanity.com/paper/2211.15595/) (2022)
- [Diffusion Models as Artists: Are we Closing the Gap between Humans and Machines?](https://mlsanity.com/paper/2301.11722/) (2023)
- [ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans](https://mlsanity.com/paper/2502.07962/) (2025)
- [Self-Attention Amortized Distributional Projection Optimization for Sliced Wasserstein Point-Cloud Reconstruction](https://mlsanity.com/paper/2301.04791/) (2023)
- [Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data](https://mlsanity.com/paper/2402.00743/) (2024)
- [KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning](https://mlsanity.com/paper/2409.12865/) (2024)
- [Why is prompting hard? Understanding prompts on binary sequence predictors](https://mlsanity.com/paper/2502.10760/) (2025)
- [Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight](https://mlsanity.com/paper/2509.24169/) (2025)
- [EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment](https://mlsanity.com/paper/2410.05938/) (2024)
- [An alternative formulation of attention pooling function in translation](https://mlsanity.com/paper/2409.00068/) (2024)
- [SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging](https://mlsanity.com/paper/2506.08297/) (2025)
- [DeMansia: Mamba Never Forgets Any Tokens](https://mlsanity.com/paper/2408.01986/) (2024)
- [Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks](https://mlsanity.com/paper/2412.00546/) (2024)
- [Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics](https://mlsanity.com/paper/2506.02212/) (2025)
- [MixFormer: Linear Transformer with Mixture of Memory Experts](https://mlsanity.com/paper/2608.09468/) (2026)

## Abstract (arXiv)

In this paper, to remedy this deficiency, we propose a Linear Attention Mechanism which is approximate to dot-product attention with much less memory and computational costs. The efficient design makes the incorporation between attention mechanisms and neural networks more flexible and versatile. Experiments conducted on semantic segmentation demonstrated the effectiveness of linear attention mechanism. Code is available at https://github.com/lironui/Linear-Attention-Mechanism.