mlsanityData to 14 Oct 2026

Demystify Transformers & Convolutions in Modern Image Deep Networks

arXiv 2211.05781 ↗First posted 10 Nov 2022cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Vision transformers have gained popularity recently, leading to the development of new vision backbones with improved features and consistent performance gains. However, these advancements are not solely attributable to novel feature transformation designs; certain benefits also arise from advanced network-level and block-level architectures. This paper aims to identify the real gains of popular convolution and attention operators through a detailed study. We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the "spatial token mixer" (STM). To facilitate an impartial comparison, we introduce a unified architecture to neutralize the impact of divergent network-level and block-level designs. Subsequently, various STMs are integrated into this unified framework for comprehensive comparative analysis. Our experiments on various tasks and an analysis of inductive bias show a significant performance boost due to advanced network-level and block-level designs, but performance differences persist among different STMs. Our detailed analysis also reveals various findings about different STMs, including effective receptive fields, invariance, and adversarial robustness tests.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

43 of 59 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2017Mask R-CNNcited by 7,277
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2017Squeeze-and-Excitation Networkscited by 4,789
31 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  2. 2014Going Deeper with Convolutionscited by 4,241
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  4. 2022A ConvNet for the 2020scited by 3,579
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2017Non-local Neural Networkscited by 2,293
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  10. 2017Deformable Convolutional Networkscited by 1,488
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2017Random Erasing Data Augmentationcited by 1,239
  13. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  14. 2016Deep Networks with Stochastic Depthcited by 979
  15. 2021Going deeper with Image Transformerscited by 764
  16. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  17. 2017Dynamic Routing Between Capsulescited by 689
  18. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 670
  19. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  20. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
  21. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  22. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  23. 2019Making Convolutional Networks Shift-Invariant Againcited by 391
  24. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  25. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  26. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  27. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  28. 2022MetaFormer Baselines for Visioncited by 132
  29. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  30. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 62
  31. 2021Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attentioncited by 36

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 3.

7 papers.

  1. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  2. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 102
  3. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 98
  4. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  5. 2023UniNeXt: Exploring A Unified Architecture for Vision Recognitioncited by 16
  6. 2023Are Large Kernels Better Teachers than Transformers for ConvNets?cited by 8
  7. 2024Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Eracited by 7