mlsanityData to 7 Oct 2026

Is Attention Better Than Matrix Decomposition?

arXiv 2109.04553 ↗First posted 9 Sep 2021cs.CV cs.LGcited by 75

Not on the map. Why?

Abstract (arXiv)

As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition (MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank recovery problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 85 of its 122 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

85 of 122 references.

References · 85

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
38 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2017Non-local Neural Networkscited by 2,290
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  7. 2017Deformable Convolutional Networkscited by 1,481
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  10. 2016Group Equivariant Convolutional Networkscited by 1,154
  11. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  12. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  13. 2019Attention is not Explanationcited by 916
  14. 2018Self-Attention Generative Adversarial Networkscited by 809
  15. 2020PCT: Point cloud transformercited by 730
  16. 2017Dynamic Routing Between Capsulescited by 689
  17. 2019Attention is not not Explanationcited by 629
  18. 2019Deep Equilibrium Modelscited by 603
  19. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  20. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  21. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  22. 2019Meta-Learning with Implicit Gradientscited by 523
  23. 2020Hopfield Networks is All You Needcited by 461
  24. 2019General E(2)-Equivariant Steerable CNNscited by 441
  25. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
  26. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  27. 2019Is Attention Interpretable?cited by 387
  28. 2014Recurrent Models of Visual Attentioncited by 344
  29. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  30. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  31. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
  32. 2018Context Encoding for Semantic Segmentationcited by 289
  33. 2021Video Transformer Networkcited by 271
  34. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  35. 2018cGANs with Projection Discriminatorcited by 253
  36. 2020Neural Architecture Search without Trainingcited by 241
  37. 2019Recurrent Independent Mechanismscited by 219
  38. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 23; 2023: 16; 2024: 13; 2025: 11; 2026: 6.

75 papers.

Sort:

By area:

  1. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  2. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  3. 2022Visual Attention Networkcited by 210
  4. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  5. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  6. 2021On Training Implicit Modelscited by 73
  7. 2022Deep Equilibrium Optical Flow Estimationcited by 63
  8. 2023One-Step Diffusion Distillation via Deep Equilibrium Modelscited by 60
  9. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  10. 2023MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal and Channel Mixingcited by 52
  11. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  12. 2021VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantizationcited by 39
38 more
  1. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
  2. 2024SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentationcited by 31
  3. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  4. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
  5. 2023One-step differentiation of iterative algorithmscited by 28
  6. 2022Transformers from an Optimization Perspectivecited by 27
  7. 2022Eliminating Gradient Conflict in Reference-based Line-Art Colorizationcited by 25
  8. 2024Context-Guided Spatial Feature Reconstruction for Efficient Semantic Segmentationcited by 25
  9. 2023ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolutioncited by 23
  10. 2022Alternating Differentiation for Optimization Layerscited by 22
  11. 2022Learning Self-Supervised Low-Rank Network for Single-Stage Weakly and Semi-Supervised Semantic Segmentationcited by 21
  12. 2022Grounded Affordance from Exocentric Viewcited by 21
  13. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  14. 2023Deep Equilibrium Diffusion Restoration with Parallel Samplingcited by 17
  15. 2023DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibriumcited by 16
  16. 2025SCSegamba: Lightweight Structure-Aware Vision Mamba for Crack Segmentation in Structurescited by 16
  17. 2024FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensingcited by 13
  18. 2021Implicit vs Unfolded Graph Neural Networkscited by 12
  19. 2022Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentationcited by 12
  20. 2022Convexifying Transformers: Improving optimization and understanding of transformer networkscited by 12
  21. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  22. 2023Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsitycited by 11
  23. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  24. 2022Softmax-free Linear Transformerscited by 9
  25. 2023Polyper: Boundary Sensitive Polyp Segmentationcited by 8
  26. 2022CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidancecited by 7
  27. 2022Computational Imaging for Machine Perception: Transferring Semantic Segmentation beyond Aberrationscited by 7
  28. 2024Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysiscited by 7
  29. 2024On Logical Extrapolation for Mazes with Recurrent and Implicit Networkscited by 7
  30. 2021Sampling Equivariant Self-attention Networks for Object Detection in Aerial Imagescited by 6
  31. 2024Gasformer: A Transformer-based Architecture for Segmenting Methane Emissions from Livestock in Optical Gas Imagingcited by 6
  32. 2024Staircase Cascaded Fusion of Lightweight Local Pattern Recognition and Long-Range Dependencies for Structural Crack Segmentationcited by 6
  33. 2022Simple Baseline for Weather Forecasting Using Spatiotemporal Context Aggregation Networkcited by 5
  34. 2022Orthogonal SVD Covariance Conditioning and Latent Disentanglementcited by 5
  35. 2023MCANet: Medical Image Segmentation with Multi-Scale Cross-Axis Attentioncited by 5
  36. 2022FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalizationcited by 4
  37. 2022Improving Covariance Conditioning of the SVD Meta-layer by Orthogonalitycited by 4
  38. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4

50 most cited of 75 All 75 (JSON)