mlsanityData to 14 Oct 2026

Rethinking Spatial Dimensions of Vision Transformers

arXiv 2103.16302 ↗First posted 30 Mar 2021cs.CVcited by 361

Not on the map. Why?

Abstract (arXiv)

Vision Transformer (ViT) extends the application range of transformers from language processing to computer vision tasks as being an alternative architecture against the existing convolutional neural networks (CNN). Since the transformer-based architecture has been innovative for computer vision modeling, the design convention towards an effective architecture has been less studied yet. From the successful design principles of CNN, we investigate the role of spatial dimension conversion and its effectiveness on transformer-based architecture. We particularly attend to the dimension reduction principle of CNNs; as the depth increases, a conventional CNN increases channel dimension and decreases spatial dimensions. We empirically show that such a spatial dimension reduction is beneficial to a transformer architecture as well, and propose a novel Pooling-based Vision Transformer (PiT) upon the original ViT model. We show that PiT achieves the improved model capability and generalization performance against ViT. Throughout the extensive experiments, we further show PiT outperforms the baseline on several tasks such as image classification, object detection, and robustness evaluation. Source codes and ImageNet models are available at https://github.com/naver-ai/pit

Key facts. Cited by 361 papers in the map (data through 14 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

40 of 47 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2016Layer Normalizationcited by 5,669
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
28 more
  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  4. 2017Non-local Neural Networkscited by 2,293
  5. 2019Searching for MobileNetV3cited by 1,994
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2018Group Normalizationcited by 1,524
  8. 2019Natural Adversarial Examplescited by 1,408
  9. 2020Designing Network Design Spacescited by 960
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  11. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  12. 2017Fine-tuning CNN Image Retrieval with No Human Annotationcited by 499
  13. 2020ResNeSt: Split-Attention Networkscited by 445
  14. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  15. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 343
  16. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  17. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  18. 2016Deep Pyramidal Residual Networkscited by 163
  19. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  20. 2016Fully Convolutional Instance-aware Semantic Segmentationcited by 123
  21. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
  22. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  23. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  24. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  25. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  26. 2020Rethinking Channel Dimensions for Efficient Model Designcited by 41
  27. 2019Positional Normalizationcited by 28
  28. 2021Rethinking Natural Adversarial Examples for Classification Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 59; 2022: 107; 2023: 63; 2024: 62; 2025: 52; 2026: 18.

361 papers.

Sort:

By area:

  1. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 866
  2. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  3. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  4. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  5. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  6. 2022DeiT III: Revenge of the ViTcited by 419
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  8. 2022How Do Vision Transformers Work?cited by 307
  9. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  10. 2024Rotary Position Embedding for Vision Transformercited by 236
  11. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  12. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
38 more
  1. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  2. 2022DaViT: Dual Attention Vision Transformerscited by 175
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  4. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 165
  5. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  6. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  7. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  8. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  9. 2022Understanding The Robustness in Vision Transformerscited by 144
  10. 2021Towards Robust Vision Transformercited by 137
  11. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 127
  12. 2021Vision Transformer for Small-Size Datasetscited by 122
  13. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  14. 2021A Survey of Visual Transformerscited by 117
  15. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  16. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
  17. 2023What Do Self-Supervised Vision Transformers Learn?cited by 93
  18. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  19. 2021P2T: Pyramid Pooling Transformer for Scene Understandingcited by 84
  20. 2021S2-MLP: Spatial-Shift MLP Architecture for Visioncited by 83
  21. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  22. 2021Causal Attention for Unbiased Visual Recognitioncited by 80
  23. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  24. 2021TransMix: Attend to Mix for Vision Transformerscited by 79
  25. 2022ObjectFormer for Image Manipulation Detection and Localizationcited by 78
  26. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 76
  27. 2022What do Vision Transformers Learn? A Visual Explorationcited by 76
  28. 2021Token Pooling in Vision Transformerscited by 69
  29. 2023SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationscited by 66
  30. 2023How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defensescited by 66
  31. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 65
  32. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  33. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  34. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  35. 2021On the Connection between Local Attention and Dynamic Depth-wise Convolutioncited by 62
  36. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  37. 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61
  38. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60

50 most cited of 361 All 361 (JSON)