mlsanityData to 7 Oct 2026

CvT: Introducing Convolutions to Vision Transformers

arXiv 2103.15808 ↗First posted 29 Mar 2021cs.CVcited by 946

Not on the map. Why?

Abstract (arXiv)

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at urlhttps://github.com/leoxiaobin/CvT.

Key facts. Cited by 946 papers in the map (data through 7 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

36 of 47 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2016Layer Normalizationcited by 5,579
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
24 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2020Point Transformercited by 1,462
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  7. 2020Pre-Trained Image Processing Transformercited by 891
  8. 2018Image Transformercited by 770
  9. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  10. 2021Transformer in Transformercited by 632
  11. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  12. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  13. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  14. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  15. 2020Are we done with ImageNet?cited by 329
  16. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  17. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  18. 2019Local Relation Networks for Image Recognitioncited by 210
  19. 2020Learning Joint Spatial-Temporal Transformations for Video Inpaintingcited by 162
  20. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  21. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  22. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  23. 2021Evolving Attention with Residual Convolutionscited by 20
  24. 2020DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Searchcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 148; 2022: 268; 2023: 205; 2024: 183; 2025: 95; 2026: 47.

946 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  4. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  6. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  7. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  8. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  9. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  10. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  11. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  12. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
38 more
  1. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  2. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  3. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  4. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 508
  5. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  6. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  7. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  8. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  9. 2022DeiT III: Revenge of the ViTcited by 417
  10. 2021Early Convolutions Help Transformers See Bettercited by 395
  11. 2021Pay Attention to MLPscited by 367
  12. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  13. 2022Vision Transformer with Deformable Attentioncited by 293
  14. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  15. 2022MixFormer: End-to-End Tracking with Iterative Mixed Attentioncited by 278
  16. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  17. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  18. 2021Global Filter Networks for Image Classificationcited by 257
  19. 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 243
  20. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  21. 2022Neighborhood Attention Transformercited by 235
  22. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  23. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  24. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  25. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  26. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 216
  27. 2022Visual Attention Networkcited by 210
  28. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  29. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  30. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  31. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  32. 2022Focal Modulation Networkscited by 186
  33. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  34. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  35. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  36. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  37. 2022DaViT: Dual Attention Vision Transformerscited by 174
  38. 2021BEVT: BERT Pretraining of Video Transformerscited by 169

50 most cited of 946 All 946 (JSON)