mlsanityData to 7 Oct 2026

Visformer: The Vision-friendly Transformer

arXiv 2104.12533 ↗First posted 26 Apr 2021cs.CVcited by 137

Not on the map. Why?

Abstract (arXiv)

The past year has witnessed the rapid development of applying the Transformer module to vision problems. While some researchers have demonstrated that Transformer-based models enjoy a favorable ability of fitting data, there are still growing number of evidences showing that these models suffer over-fitting especially when the training data is limited. This paper offers an empirical study by performing step-by-step operations to gradually transit a Transformer-based model to a convolution-based model. The results we obtain during the transition process deliver useful messages for improving visual recognition. Based on these observations, we propose a new architecture named Visformer, which is abbreviated from the `Vision-friendly Transformer'. With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy, and the advantage becomes more significant when the model complexity is lower or the training set is smaller. The code is available at https://github.com/danczs/Visformer.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

48 of 67 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2020End-to-End Object Detection with Transformerscited by 7,384
36 more
  1. 2017Mask R-CNNcited by 7,259
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Layer Normalizationcited by 5,579
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2017Non-local Neural Networkscited by 2,290
  13. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  14. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  15. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  16. 2018Group Normalizationcited by 1,514
  17. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  18. 2017Random Erasing Data Augmentationcited by 1,235
  19. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  20. 2016Deep Networks with Stochastic Depthcited by 973
  21. 2020Designing Network Design Spacescited by 954
  22. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  23. 2020Pre-Trained Image Processing Transformercited by 891
  24. 2021Going deeper with Image Transformerscited by 759
  25. 2013Network In Networkcited by 600
  26. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  27. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  28. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  29. 2019Attention Augmented Convolutional Networkscited by 363
  30. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  31. 2020Exploring Self-attention for Image Recognitioncited by 313
  32. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  33. 2019Local Relation Networks for Image Recognitioncited by 210
  34. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  35. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  36. 2020Neural Architecture Search for Lightweight Non-Local Networkscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 30; 2023: 31; 2024: 24; 2025: 24; 2026: 10.

137 papers.

Sort:

By area:

  1. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  2. 2021Early Convolutions Help Transformers See Bettercited by 395
  3. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  4. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  5. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  6. 2022DaViT: Dual Attention Vision Transformerscited by 174
  7. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  8. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  9. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  10. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  11. 2023FasterViT: Fast Vision Transformers with Hierarchical Attentioncited by 65
  12. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
38 more
  1. 2023Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Reviewcited by 57
  2. 2023Boosting Adversarial Transferability by Achieving Flat Local Maximacited by 48
  3. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  4. 2022MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Modelscited by 47
  5. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  6. 2021Are Vision Transformers Robust to Patch Perturbations?cited by 39
  7. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  8. 2022Affine Medical Image Registration with Coarse-to-Fine Vision Transformercited by 35
  9. 2022Towards Data-Efficient Detection Transformerscited by 34
  10. 2023Reversible Vision Transformerscited by 33
  11. 2021Scaled ReLU Matters for Training Vision Transformerscited by 29
  12. 2023Semantic Prompt for Few-Shot Image Recognitioncited by 29
  13. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28
  14. 2021Towards End-to-End Image Compression and Analysis with Transformerscited by 26
  15. 2022Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognitioncited by 26
  16. 2024Learning to Transform Dynamically for Better Adversarial Transferabilitycited by 23
  17. 2021Building extraction with vision transformercited by 22
  18. 2022Self-Promoted Supervision for Few-Shot Transformercited by 22
  19. 2022CNN Filter DB: An Empirical Investigation of Trained Convolutional Filterscited by 20
  20. 2022Boosting vision transformers for image retrievalcited by 19
  21. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19
  22. 2021ELSA: Enhanced Local Self-Attention for Vision Transformercited by 18
  23. 2022Part-based Face Recognition with Vision Transformerscited by 18
  24. 2023Distilling Efficient Vision Transformers from CNNs for Semantic Segmentationcited by 18
  25. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  26. 2023Multi-event Video-Text Retrievalcited by 15
  27. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  28. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
  29. 2024Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasetscited by 14
  30. 2024LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognitioncited by 13
  31. 2022Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Surveycited by 12
  32. 2023Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attentioncited by 12
  33. 2024HIRI-ViT: Scaling Vision Transformer with High Resolution Inputscited by 12
  34. 2025Vision Transformers with Self-Distilled Registerscited by 12
  35. 2022GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagationcited by 11
  36. 2023Make A Long Image Short: Adaptive Token Length for Vision Transformerscited by 10
  37. 2022What Makes for Good Tokenizers in Vision Transformer?cited by 9
  38. 2023Map-and-Conquer: Energy-Efficient Mapping of Dynamic Neural Nets onto Heterogeneous MPSoCscited by 9

50 most cited of 137 All 137 (JSON)