# Breaking the Low-Rank Dilemma of Linear Attention

arXiv: https://arxiv.org/abs/2411.07635 · HTML: https://mlsanity.com/paper/2411.07635/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 76 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)

## Built on by (most cited first, 30 of 31)

- [SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention](https://mlsanity.com/paper/2509.24006/) (2025)
- [PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.01077/) (2026)
- [Rectifying Magnitude Neglect in Linear Attention](https://mlsanity.com/paper/2507.00698/) (2025)
- [Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention](https://mlsanity.com/paper/2511.06294/) (2025)
- [FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling](https://mlsanity.com/paper/2603.06199/) (2026)
- [MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head](https://mlsanity.com/paper/2601.07832/) (2026)
- [DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling](https://mlsanity.com/paper/2505.11196/) (2025)
- [The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures](https://mlsanity.com/paper/2510.05364/) (2025)
- [ViT-AdaLA: Adapting Vision Transformers with Linear Attention](https://mlsanity.com/paper/2603.16063/) (2026)
- [Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion](https://mlsanity.com/paper/2605.16579/) (2026)
- [RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers](https://mlsanity.com/paper/2605.20659/) (2026)
- [S2AFormer: Strip Self-Attention for Efficient Vision Transformer](https://mlsanity.com/paper/2505.22195/) (2025)
- [SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attention](https://mlsanity.com/paper/2509.12817/) (2025)
- [VMonarch: Efficient Video Diffusion Transformers with Structured Attention](https://mlsanity.com/paper/2601.22275/) (2026)
- [State Rank Dynamics in Linear Attention LLMs](https://mlsanity.com/paper/2602.02195/) (2026)
- [HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling](https://mlsanity.com/paper/2505.20836/) (2025)
- [Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attention](https://mlsanity.com/paper/2506.21137/) (2025)
- [Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention](https://mlsanity.com/paper/2507.07247/) (2025)
- [GraphTARIF: Linear Graph Transformer with Augmented Rank and Improved Focus](https://mlsanity.com/paper/2510.10631/) (2025)
- [EFlow: Fast Few-Step Video Generator Training from Scratch via Efficient Solution Flow](https://mlsanity.com/paper/2603.27086/) (2026)
- [UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification](https://mlsanity.com/paper/2605.06221/) (2026)
- [SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation](https://mlsanity.com/paper/2607.21553/) (2026)
- [RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models](https://mlsanity.com/paper/2511.18380/) (2025)
- [HLA: Hadamard Linear Attention](https://mlsanity.com/paper/2602.12128/) (2026)
- [Vision Non-Causal Trapezoidal Mamba: Eliminating Directional Scanning in Vision SSMs with Second-Order Dynamics](https://mlsanity.com/paper/2607.03589/) (2026)
- [Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention](https://mlsanity.com/paper/2608.30129/) (2026)
- [Region-Aware Deformable Convolutions](https://mlsanity.com/paper/2509.15436/) (2025)
- [Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency](https://mlsanity.com/paper/2510.20092/) (2025)
- [Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention](https://mlsanity.com/paper/2603.00175/) (2026)
- [Representative Attention For Vision Transformers](https://mlsanity.com/paper/2605.14913/) (2026)

## Abstract (arXiv)

The Softmax attention mechanism in Transformer models is notoriously computationally expensive, particularly due to its quadratic complexity, posing significant challenges in vision applications. In contrast, linear attention provides a far more efficient solution by reducing the complexity to linear levels. However, compared to Softmax attention, linear attention often experiences significant performance degradation. Our experiments indicate that this performance drop is due to the low-rank nature of linear attention's feature map, which hinders its ability to adequately model complex spatial information. In this paper, to break the low-rank dilemma of linear attention, we conduct rank analysis from two perspectives: the KV buffer and the output features. Consequently, we introduce Rank-Augmented Linear Attention (RALA), which rivals the performance of Softmax attention while maintaining linear complexity and high efficiency. Based on RALA, we construct the Rank-Augmented Vision Linear Transformer (RAVLT). Extensive experiments demonstrate that RAVLT achieves excellent performance across various vision tasks. Specifically, without using any additional labels, data, or supervision during training, RAVLT achieves an 84.4% Top-1 accuracy on ImageNet-1k with only 26M parameters and 4.6G FLOPs. This result significantly surpasses previous linear attention mechanisms, fully illustrating the potential of RALA. Code will be available at https://github.com/qhfan/RALA.