mlsanityData to 7 Oct 2026

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

arXiv 2201.10801 ↗First posted 26 Jan 2022cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Attention mechanism has been widely believed as the key to success of vision transformers (ViTs), since it provides a flexible and powerful way to model spatial relationships. However, is the attention mechanism truly an indispensable part of ViT? Can it be replaced by some other alternatives? To demystify the role of attention mechanism, we simplify it into an extremely simple case: ZERO FLOP and ZERO parameter. Concretely, we revisit the shift operation. It does not contain any parameter or arithmetic calculation. The only operation is to exchange a small portion of the channels between neighboring features. Based on this simple operation, we construct a new backbone network, namely ShiftViT, where the attention layers in ViT are substituted by shift operations. Surprisingly, ShiftViT works quite well in several mainstream tasks, e.g., classification, detection, and segmentation. The performance is on par with or even better than the strong baseline Swin Transformer. These results suggest that the attention mechanism might not be the vital factor that makes ViT successful. It can be even replaced by a zero-parameter operation. We should pay more attentions to the remaining parts of ViT in the future work. Code is available at github.com/microsoft/SPACH.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 24 of its 26 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

24 of 26 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
12 more
  1. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  2. 2021Pay Attention to MLPscited by 367
  3. 2020Exploring Self-attention for Image Recognitioncited by 313
  4. 2017Relation Networks for Object Detectioncited by 305
  5. 2019Local Relation Networks for Image Recognitioncited by 210
  6. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  7. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101
  8. 2017Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutionscited by 93
  9. 2021S2-MLP: Spatial-Shift MLP Architecture for Visioncited by 83
  10. 2019All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classificationcited by 38
  11. 2018Constructing Fast Network through Deconstruction of Convolutioncited by 24
  12. 2021Local-to-Global Self-Attention in Vision Transformerscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 10; 2024: 8; 2025: 5; 2026: 1.

30 papers.

Sort:

By area:

  1. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 129
  2. 2022A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shiftcited by 68
  3. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  4. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 27
  5. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  6. 2022Genie: Show Me the Data for Quantizationcited by 17
  7. 2023ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformercited by 15
  8. 2023SPANet: Frequency-balancing Token Mixer using Spectral Pooling Aggregation Modulationcited by 14
  9. 2024KANICE: Kolmogorov-Arnold Networks with Interactive Convolutional Elementscited by 12
  10. 2022GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagationcited by 11
  11. 2024Infrared Image Super-Resolution via Lightweight Information Split Networkcited by 10
  12. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
18 more
  1. 2023Aperture Diffraction for Compact Snapshot Spectral Imagingcited by 7
  2. 2022Active Token Mixercited by 6
  3. 2023Convolution and Attention Mixer for Synthetic Aperture Radar Image Change Detectioncited by 4
  4. 2024ShiftwiseConv: Small Convolutional Kernel with Large Kernel Effectcited by 4
  5. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  6. 2023S3Aug: Segmentation, Sampling, and Shift for Action Recognitioncited by 3
  7. 2025LipShiFT: A Certifiably Robust Shift-based Vision Transformercited by 3
  8. 2023Towards an Effective and Efficient Transformer for Rain-by-snow Weather Removalcited by 2
  9. 2022SPIN: An Empirical Evaluation on Sharing Parameters of Isotropic Networkscited by 1
  10. 2023ReBotNet: Fast Real-time Video Enhancementcited by 1
  11. 2023Transforming the Output of Generative Pre-trained Transformer: The Influence of the PGI Framework on Attention Dynamicscited by 1
  12. 2025An Explainable Transformer Model for Alzheimer's Disease Detection Using Retinal Imagingcited by 1
  13. 2024Mixer is more than just a modelcited by 0
  14. 2024DCNN: Dual Cross-current Neural Networks Realized Using An Interactive Deep Learning Discriminator for Fine-grained Objectscited by 0
  15. 2024MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformerscited by 0
  16. 2025Spectral-Adaptive Modulation Networks for Visual Perceptioncited by 0
  17. 2025DeepTraverse: A Depth-First Search Inspired Network for Algorithmic Visual Understandingcited by 0
  18. 2026ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentationcited by 0