mlsanityData to 7 Oct 2026

Skip-Attention: Improving Vision Transformers by Paying Less Attention

arXiv 2301.02240 ↗First posted 5 Jan 2023cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

This work aims to improve the efficiency of vision transformers (ViT). While ViTs use computationally expensive self-attention operations in every layer, we identify that these operations are highly correlated across layers -- a key redundancy that causes unnecessary computations. Based on this observation, we propose SkipAt, a method to reuse self-attention computation from preceding layers to approximate attention at one or more subsequent layers. To ensure that reusing self-attention blocks across layers does not degrade the performance, we introduce a simple parametric function, which outperforms the baseline transformer's performance while running computationally faster. We show the effectiveness of our method in image classification and self-supervised learning on ImageNet-1K, semantic segmentation on ADE20K, image denoising on SIDD, and video denoising on DAVIS. We achieve improved throughput at the same-or-higher accuracy levels in all these tasks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 56 of its 70 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

56 of 70 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016Layer Normalizationcited by 5,579
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
38 more
  1. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  2. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  5. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  6. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  8. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  9. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  10. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  12. 2021Multi-Stage Progressive Image Restorationcited by 813
  13. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  14. 2019Panoptic Feature Pyramid Networkscited by 644
  15. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  16. 2021Transformer in Transformercited by 632
  17. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 572
  18. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  19. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  20. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  21. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  22. 2018Video Action Transformer Networkcited by 316
  23. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  24. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  25. 2022VRT: A Video Restoration Transformercited by 219
  26. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  27. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  28. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  29. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  30. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  31. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  32. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  33. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  34. 2019FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimationcited by 128
  35. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  36. 2021The Efficiency Misnomercited by 98
  37. 2021Multi-Scale High-Resolution Vision Transformer for Semantic Segmentationcited by 92
  38. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 7; 2026: 3.

23 papers.

Sort:

By area:

  1. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  2. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  3. 2023ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detectioncited by 76
  4. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  5. 2025SageAttention2++: A More Efficient Implementation of SageAttention2cited by 21
  6. 2023Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?cited by 16
  7. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 16
  8. 2024Sharing Key Semantics in Transformer Makes Efficient Image Restorationcited by 14
  9. 2025Efficient Degradation-agnostic Image Restoration via Channel-Wise Functional Decomposition and Manifold Regularizationcited by 8
  10. 2023Efficiency 360: Efficient Vision Transformerscited by 6
  11. 2024Advancing Supervised Local Learning Beyond Classification with Long-term Feature Bankcited by 6
  12. 2024Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inferencecited by 6
11 more
  1. 2024Linearly-evolved Transformer for Pan-sharpeningcited by 5
  2. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  3. 2025Enhancing Layer Attention Efficiency through Pruning Redundant Retrievalscited by 3
  4. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  5. 2024Replacement Learning: Training Vision Tasks with Fewer Learnable Parameterscited by 1
  6. 2024TOAST: Transformer Optimization using Adaptive and Simple Transformationscited by 1
  7. 2024Memory Efficient Matting with Adaptive Token Routingcited by 0
  8. 2025Network of Theseus (like the ship)cited by 0
  9. 2026Replacement Learning: Training Neural Networks with Fewer Parameterscited by 0
  10. 2026RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generationcited by 0
  11. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0