mlsanityData to 7 Oct 2026

Going deeper with Image Transformers

arXiv 2103.17239 ↗First posted 31 Mar 2021cs.CVcited by 759

Not on the map. Why?

Abstract (arXiv)

Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of image transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two transformers architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for instance we obtain 86.5% top-1 accuracy on Imagenet when training with no external data, we thus attain the current SOTA with less FLOPs and parameters. Moreover, our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data. We share our code and models.

Key facts. Cited by 759 papers in the map (data through 7 Oct 2026). Builds on 65 of its 81 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

65 of 81 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  13. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  14. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  15. 2017Deformable Convolutional Networkscited by 1,481
  16. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  17. 2016Deep Networks with Stochastic Depthcited by 973
  18. 2020Designing Network Design Spacescited by 954
  19. 2018Image Transformercited by 770
  20. 2021Transformer in Transformercited by 632
  21. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  22. 2017Non-Autoregressive Neural Machine Translationcited by 457
  23. 2020ResNeSt: Split-Attention Networkscited by 445
  24. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  25. 2019Selective Kernel Networkscited by 401
  26. 2020Meta Pseudo Labelscited by 372
  27. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  28. 2019Attention Augmented Convolutional Networkscited by 363
  29. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  30. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  31. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  32. 2020Are we done with ImageNet?cited by 329
  33. 2020Exploring Self-attention for Image Recognitioncited by 313
  34. 2019Adversarial Examples Improve Image Recognitioncited by 283
  35. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  36. 2015Highway Networkscited by 257
  37. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  38. 2019Fixing the train-test resolution discrepancycited by 209

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 107; 2022: 185; 2023: 157; 2024: 110; 2025: 100; 2026: 100.

759 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  3. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  4. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  5. 2021Scaling Vision Transformerscited by 993
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  9. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
38 more
  1. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  2. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  3. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  4. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  5. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  6. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  7. 2022DeiT III: Revenge of the ViTcited by 417
  8. 2021Early Convolutions Help Transformers See Bettercited by 395
  9. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  10. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  12. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  13. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  14. 2024What matters when building vision-language models?cited by 321
  15. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  16. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  17. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  18. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  19. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  20. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  21. 2021Global Filter Networks for Image Classificationcited by 257
  22. 2023A Cookbook of Self-Supervised Learningcited by 248
  23. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  24. 2022Neighborhood Attention Transformercited by 235
  25. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  26. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  27. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  28. 2021FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flowscited by 225
  29. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  30. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  31. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  32. 2022Visual Attention Networkcited by 210
  33. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  34. 2022Patches Are All You Need?cited by 201
  35. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  36. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  37. 2021Are Transformers More Robust Than CNNs?cited by 187
  38. 2022Focal Modulation Networkscited by 186

50 most cited of 759 All 759 (JSON)