mlsanityData to 14 Oct 2026

Making Vision Transformers Truly Shift-Equivariant

arXiv 2305.16316 ↗First posted 25 May 2023cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

For computer vision, Vision Transformers (ViTs) have become one of the go-to deep net architectures. Despite being inspired by Convolutional Neural Networks (CNNs), ViTs' output remains sensitive to small spatial shifts in the input, i.e., not shift invariant. To address this shortcoming, we introduce novel data-adaptive designs for each of the modules in ViTs, such as tokenization, self-attention, patch merging, and positional encoding. With our proposed modules, we achieve true shift-equivariance on four well-established ViTs, namely, Swin, SwinV2, CvT, and MViTv2. Empirically, we evaluate the proposed adaptive models on image classification and semantic segmentation tasks. These models achieve competitive performance across three different datasets while maintaining 100% shift consistency.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 51 of its 65 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

51 of 65 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  10. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  11. 2017Deep Setscited by 1,421
  12. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
38 more
  1. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  2. 2016Group Equivariant Convolutional Networkscited by 1,165
  3. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  4. 2021Transformers in Vision: A Surveycited by 1,013
  5. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  6. 2021Multiscale Vision Transformerscited by 897
  7. 2020A Survey on Visual Transformercited by 654
  8. 2013Network In Networkcited by 605
  9. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  10. 2018Spherical CNNscited by 456
  11. 2019General E(2)-Equivariant Steerable CNNscited by 446
  12. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  13. 2018Invariant and Equivariant Graph Networkscited by 346
  14. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  15. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 267
  16. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  17. 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 171
  18. 2017Equivariance Through Parameter-Sharingcited by 141
  19. 2020On Learning Sets of Symmetric Elementscited by 98
  20. 2020Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphscited by 92
  21. 2018Deep Models of Interactions Across Setscited by 69
  22. 2021Improved OOD Generalization via Adversarial Training and Pre-trainingcited by 66
  23. 2020Attentive Group Equivariant Convolutional Networkscited by 63
  24. 2020Delving Deeper into Anti-aliasing in ConvNetscited by 60
  25. 2020Truly shift-invariant convolutional neural networkscited by 53
  26. 2019PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learningcited by 47
  27. 2017Geometric robustness of deep networks: analysis and improvementcited by 45
  28. 2021Learning Partial Equivariances from Datacited by 45
  29. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  30. 2022SpeqNets: Sparsity-aware Permutation-equivariant Graph Networkscited by 37
  31. 2019Chirality Nets for Human Pose Regressioncited by 31
  32. 2021Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learningcited by 18
  33. 2022Equivariance Discovery by Learned Parameter-Sharingcited by 18
  34. 2022Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networkscited by 18
  35. 2023Image to Sphere: Learning Equivariant Features for Efficient Pose Predictioncited by 18
  36. 2021Equivariant Networks for Pixelized Spherescited by 16
  37. 2023Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activationscited by 11
  38. 2019Building Deep, Equivariant Capsule Networkscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 8; 2026: 4.

16 papers.

Sort:

By area:

  1. 2023Truly Scale-Equivariant Deep Nets with Fourier Layerscited by 16
  2. 2024Improving Shift Invariance in Convolutional Neural Networks with Translation Invariant Polyphase Samplingcited by 6
  3. 2025Enhancing Image Restoration Transformer via Adaptive Translation Equivariancecited by 5
  4. 2025Group Downsampling with Equivariant Anti-aliasingcited by 3
  5. 2025Quick ViTs: Speeding up Vision Transformers through Equivariancecited by 3
  6. 2026Vanilla Group Equivariant Vision Transformer: Simple and Effectivecited by 3
  7. 2023Efficient Rotation Invariance in Deep Neural Networks through Artificial Mental Rotationcited by 2
  8. 2025Local Scale Equivariance with Latent Deep Equilibrium Canonicalizercited by 2
  9. 2025CLIPSym: Delving into Symmetry Detection with CLIPcited by 2
  10. 2025Alias-Free ViT: Fractional Shift Invariance via Linear Attentioncited by 2
  11. 2026RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetrycited by 1
  12. 2024Lost in Translation: Modern Neural Networks Still Struggle With Small Realistic Image Transformationscited by 0
4 more
  1. 2025Towards Foundation Models for Cryo-ET Subtomogram Analysiscited by 0
  2. 2025Shift-Equivariant Complex-Valued Convolutional Neural Networkscited by 0
  3. 2026Tunable Soft Equivariance with Guaranteescited by 0
  4. 2026PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuningcited by 0