mlsanityData to 14 Oct 2026

Always Skip Attention

arXiv 2505.01996 ↗First posted 4 May 2025cs.LG cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

We highlight a curious empirical result within modern Vision Transformers (ViTs). Specifically, self-attention catastrophically fails to train unless it is used in conjunction with a skip connection. This is in contrast to other elements of a ViT that continue to exhibit good performance (albeit suboptimal) when skip connections are removed. Further, we show that this critical dependence on skip connections is a relatively new phenomenon, with previous deep architectures (eg, CNNs) exhibiting good performance in their absence. In this paper, we theoretically characterize that the self-attention mechanism is fundamentally ill-conditioned and is, therefore, uniquely dependent on skip connections for regularization. Additionally, we propose Token Graying -- a simple yet effective complement (to skip connections) that further improves the condition of input tokens. We validate our approach in both supervised and self-supervised training methods.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

26 of 38 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  10. 2021Going deeper with Image Transformerscited by 764
  11. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  12. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
14 more
  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  2. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  3. 2022Patches Are All You Need?cited by 201
  4. 2023PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networkscited by 64
  5. 2024Theory, Analysis, and Best Practices for Sigmoid Self-Attentioncited by 64
  6. 2020Stable ResNetcited by 49
  7. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  8. 2024A Decade's Battle on Dataset Bias: Are We There Yet?cited by 47
  9. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  10. 2025RandLoRA: Full-rank parameter-efficient fine-tuning of large modelscited by 35
  11. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  12. 2024Pair-VPR: Place-Aware Pre-training and Contrastive Pair Classification for Visual Place Recognition with Vision Transformerscited by 15
  13. 2023Pre-training with Random Orthogonal Projection Image Modelingcited by 11
  14. 2024Weight Conditioning for Smooth Optimization of Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025Cutting the Skip: Training Residual-Free Transformerscited by 8
  2. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  3. 2025Structured Initialization for Vision Transformerscited by 6
  4. 2025Enhancing Transformers Through Conditioned Embedded Tokenscited by 3
  5. 2025SineProject: Machine Unlearning for Stable Vision Language Alignmentcited by 3
  6. 2026Spectral Conditioning of Attention Improves Transformer Performancecited by 3
  7. 2025Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformerscited by 2
  8. 2026Beyond Linearity in Attention Projections: The Case for Nonlinear Queriescited by 2
  9. 2026Orthogonal Self-Attentioncited by 1
  10. 2026Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learningcited by 1
  11. 2026Can an MLP Absorb Its Own Skip Connection Exactly?cited by 0
  12. 2026Conditioned Initialization for Attentioncited by 0
1 more
  1. 2026Should We Skip Diffusion?cited by 0