# Always Skip Attention

arXiv: https://arxiv.org/abs/2505.01996 · HTML: https://mlsanity.com/paper/2505.01996/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 4 May 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [Patches Are All You Need?](https://mlsanity.com/paper/2201.09792/) (2022)
- [PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks](https://mlsanity.com/paper/2307.11833/) (2023)
- [Theory, Analysis, and Best Practices for Sigmoid Self-Attention](https://mlsanity.com/paper/2409.04431/) (2024)
- [Stable ResNet](https://mlsanity.com/paper/2010.12859/) (2020)
- [The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit](https://mlsanity.com/paper/2306.17759/) (2023)
- [A Decade's Battle on Dataset Bias: Are We There Yet?](https://mlsanity.com/paper/2403.08632/) (2024)
- [Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation](https://mlsanity.com/paper/2302.10322/) (2023)
- [RandLoRA: Full-rank parameter-efficient fine-tuning of large models](https://mlsanity.com/paper/2502.00987/) (2025)
- [Efficient Learning With Sine-Activated Low-rank Matrices](https://mlsanity.com/paper/2403.19243/) (2024)
- [Pair-VPR: Place-Aware Pre-training and Contrastive Pair Classification for Visual Place Recognition with Vision Transformers](https://mlsanity.com/paper/2410.06614/) (2024)
- [Pre-training with Random Orthogonal Projection Image Modeling](https://mlsanity.com/paper/2310.18737/) (2023)
- [Weight Conditioning for Smooth Optimization of Neural Networks](https://mlsanity.com/paper/2409.03424/) (2024)

## Built on by (most cited first, 13 of 13)

- [Cutting the Skip: Training Residual-Free Transformers](https://mlsanity.com/paper/2510.00345/) (2025)
- [Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse](https://mlsanity.com/paper/2505.16284/) (2025)
- [Structured Initialization for Vision Transformers](https://mlsanity.com/paper/2505.19985/) (2025)
- [Enhancing Transformers Through Conditioned Embedded Tokens](https://mlsanity.com/paper/2505.12789/) (2025)
- [SineProject: Machine Unlearning for Stable Vision Language Alignment](https://mlsanity.com/paper/2511.18444/) (2025)
- [Spectral Conditioning of Attention Improves Transformer Performance](https://mlsanity.com/paper/2603.07162/) (2026)
- [Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers](https://mlsanity.com/paper/2510.23912/) (2025)
- [Beyond Linearity in Attention Projections: The Case for Nonlinear Queries](https://mlsanity.com/paper/2603.13381/) (2026)
- [Orthogonal Self-Attention](https://mlsanity.com/paper/2602.05996/) (2026)
- [Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning](https://mlsanity.com/paper/2607.17673/) (2026)
- [Can an MLP Absorb Its Own Skip Connection Exactly?](https://mlsanity.com/paper/2604.23705/) (2026)
- [Conditioned Initialization for Attention](https://mlsanity.com/paper/2609.07086/) (2026)
- [Should We Skip Diffusion?](https://mlsanity.com/paper/2610.07002/) (2026)

## Abstract (arXiv)

We highlight a curious empirical result within modern Vision Transformers (ViTs). Specifically, self-attention catastrophically fails to train unless it is used in conjunction with a skip connection. This is in contrast to other elements of a ViT that continue to exhibit good performance (albeit suboptimal) when skip connections are removed. Further, we show that this critical dependence on skip connections is a relatively new phenomenon, with previous deep architectures (eg, CNNs) exhibiting good performance in their absence. In this paper, we theoretically characterize that the self-attention mechanism is fundamentally ill-conditioned and is, therefore, uniquely dependent on skip connections for regularization. Additionally, we propose Token Graying -- a simple yet effective complement (to skip connections) that further improves the condition of input tokens. We validate our approach in both supervised and self-supervised training methods.