mlsanityData to 7 Oct 2026

PolaFormer: Polarity-aware Linear Attention for Vision Transformers

arXiv 2501.15061 ↗First posted 25 Jan 2025cs.CV cs.AIcited by 30

Not on the map. Why?

Abstract (arXiv)

Linear attention has emerged as a promising alternative to softmax-based attention, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence length. However, the non-negative constraint on feature maps and the relaxed exponential function used in approximation lead to significant information loss compared to the original query-key dot products, resulting in less discriminative attention maps with higher entropy. To address the missing interactions driven by negative values in query-key pairs, we propose a polarity-aware linear attention mechanism that explicitly models both same-signed and opposite-signed query-key interactions, ensuring comprehensive coverage of relational information. Furthermore, to restore the spiky properties of attention maps, we provide a theoretical analysis proving the existence of a class of element-wise functions (with positive first and second derivatives) that can reduce entropy in the attention distribution. For simplicity, and recognizing the distinct contributions of each dimension, we employ a learnable power function for rescaling, allowing strong and weak attention signals to be effectively separated. Extensive experiments demonstrate that the proposed PolaFormer improves performance on various vision tasks, enhancing both expressiveness and efficiency by up to 4.6%.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

35 of 47 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2017Mask R-CNNcited by 7,259
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  12. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
23 more
  1. 2024VMamba: Visual State Space Modelcited by 955
  2. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  3. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  5. 2019Panoptic Feature Pyramid Networkscited by 644
  6. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  7. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  8. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  9. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  10. 2015Particular object retrieval with integral max-pooling of CNN activationscited by 160
  11. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  12. 2022On The Computational Complexity of Self-Attentioncited by 137
  13. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  14. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  15. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  16. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  17. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  18. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  19. 2023PolySketchFormer: Fast Transformers via Sketching Polynomial Kernelscited by 33
  20. 2022Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inferencecited by 28
  21. 2024CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applicationscited by 12
  22. 2023SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computerscited by 9
  23. 2020Revisiting Linformer with a modified self-attention with linear complexitycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 20.

30 papers.

Sort:

By area:

  1. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  2. 2025LOTFormer: Doubly-Stochastic Linear Attention via Low-Rank Optimal Transportcited by 4
  3. 2025Vision Transformers are Circulant Attention Learnerscited by 4
  4. 2025SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attentioncited by 3
  5. 2026LaplacianFormer:Rethinking Linear Attention with Laplacian Kernelcited by 3
  6. 2025Norm×Direction: Restoring the Missing Query Norm in Vision Linear Attentioncited by 2
  7. 2026Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretrainingcited by 2
  8. 2025GraphTARIF: Linear Graph Transformer with Augmented Rank and Improved Focuscited by 1
  9. 2025UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attentioncited by 1
  10. 2026Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weightscited by 1
  11. 2026HLA: Hadamard Linear Attentioncited by 1
  12. 2026Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compressioncited by 1
18 more
  1. 2026AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Modelscited by 1
  2. 2026QUEST: A robust attention formulation using query-modulated spherical attentioncited by 1
  3. 2025Understanding Differential Transformer Unchains Pretrained Self-Attentionscited by 0
  4. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrogramscited by 0
  5. 2025LUNA: Linear Universal Neural Attention with Generalization Guaranteescited by 0
  6. 2026Softmax Linear Attention: Reclaiming Global Competitioncited by 0
  7. 2026MirrorLA: Reflecting Feature Map for Vision Linear Attentioncited by 0
  8. 2026Polarized Direct Cross-Attention Message Passing in GNNs for Machinery Fault Diagnosiscited by 0
  9. 2026OSA: Echocardiography Video Segmentation via Orthogonalized State Update and Anatomical Prior-aware Feature Enhancementcited by 0
  10. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0
  11. 2026Representative Attention For Vision Transformerscited by 0
  12. 2026HierEdit: Region-Aware Hierarchical Diffusion for Efficient High-Resolution Editingcited by 0
  13. 2026Head-Aware Key-Value Compression for Efficient Autoregressive Image Generationcited by 0
  14. 2026JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Searchcited by 0
  15. 2026DnA: Denoising Attention for Visual Taskscited by 0
  16. 2026Flexformer: Flexible Linear Transformer with Learnable Attention Kernelcited by 0
  17. 2026Inhibited Self-Attention: Sharpening Focus in Vision Transformerscited by 0
  18. 2026MixFormer: Linear Transformer with Mixture of Memory Expertscited by 0