mlsanityData to 14 Oct 2026

Early Convolutions Help Transformers See Better

arXiv 2106.14881 ↗First posted 28 Jun 2021cs.CVcited by 396

Not on the map. Why?

Abstract (arXiv)

Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p p*p convolution (p=16 by default) applied to the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers in neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3*3 convolutions. While the vast majority of computation in the two ViT designs is identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ~1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum of model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k to ImageNet-21k). These findings lead us to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared to the original ViT model design.

Key facts. Cited by 396 papers in the map (data through 14 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 28 Jun 2021.

Builds on

45 of 54 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2017Mask R-CNNcited by 7,277
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
33 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  6. 2017Non-local Neural Networkscited by 2,293
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2017Random Erasing Data Augmentationcited by 1,239
  12. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  13. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  14. 2020Designing Network Design Spacescited by 960
  15. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  16. 2021Multiscale Vision Transformerscited by 897
  17. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  18. 2021Going deeper with Image Transformerscited by 764
  19. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  20. 2021ImageNet-21K Pretraining for the Massescited by 613
  21. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  22. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  23. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  24. 2020Exploring Self-attention for Image Recognitioncited by 313
  25. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  26. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  27. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  28. 2021Visformer: The Vision-friendly Transformercited by 137
  29. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  30. 2019On Network Design Spaces for Visual Recognitioncited by 68
  31. 2021Fast and Accurate Model Scalingcited by 46
  32. 2019Augment your batch: better training with larger batchescited by 37
  33. 2020FBNetV3: Joint Architecture-Recipe Search using Predictor Pretrainingcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 42; 2022: 127; 2023: 95; 2024: 79; 2025: 30; 2026: 23.

396 papers.

Sort:

By area:

  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  5. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  7. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  8. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  9. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  10. 2022Activating More Pixels in Image Super-Resolution Transformercited by 438
  11. 2022DeiT III: Revenge of the ViTcited by 419
  12. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
38 more
  1. 2022How Do Vision Transformers Work?cited by 307
  2. 2022Vision Transformer with Deformable Attentioncited by 295
  3. 2022Spikformer: When Spiking Neural Network Meets Transformercited by 271
  4. 2022Vision Transformers for Single Image Dehazingcited by 260
  5. 2022Masked Visual Pre-training for Motor Controlcited by 248
  6. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  7. 2021Mobile-Former: Bridging MobileNet and Transformercited by 235
  8. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 234
  9. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  10. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 218
  11. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  12. 2022Patches Are All You Need?cited by 201
  13. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 174
  14. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  15. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 171
  16. 2022Masked World Models for Visual Controlcited by 170
  17. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  18. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  19. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  20. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 149
  21. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  22. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  23. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  24. 2021PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformercited by 120
  25. 2022DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimationcited by 118
  26. 2021A Survey of Visual Transformerscited by 117
  27. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  28. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  29. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  30. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  31. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
  32. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 109
  33. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  34. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  35. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 103
  36. 2022Inception Transformercited by 99
  37. 2023Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolationcited by 98
  38. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95

50 most cited of 396 All 396 (JSON)