mlsanityData to 7 Oct 2026

CMT: Convolutional Neural Networks Meet Vision Transformers

arXiv 2107.06263 ↗First posted 13 Jul 2021cs.CVcited by 310

Not on the map. Why?

Abstract (arXiv)

Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image. However, there are still gaps in both performance and computational cost between transformers and existing convolutional neural networks (CNNs). In this paper, we aim to address this issue and develop a network that can outperform not only the canonical transformers, but also the high-performance convolutional models. We propose a new transformer based hybrid network by taking advantage of transformers to capture long-range dependencies, and of CNNs to model local features. Furthermore, we scale it to obtain a family of models, called CMTs, obtaining much better accuracy and efficiency than previous convolution and transformer based models. In particular, our CMT-S achieves 83.5% top-1 accuracy on ImageNet, while being 14x and 2x smaller on FLOPs than the existing DeiT and EfficientNet, respectively. The proposed CMT-S also generalizes well on CIFAR10 (99.2%), CIFAR100 (91.7%), Flowers (98.7%), and other challenging vision datasets such as COCO (44.3% mAP), with considerably less computational cost.

Key facts. Cited by 310 papers in the map (data through 7 Oct 2026). Builds on 56 of its 73 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

56 of 73 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Mask R-CNNcited by 7,259
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2016Densely Connected Convolutional Networkscited by 6,691
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2016Layer Normalizationcited by 5,579
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2014Going Deeper with Convolutionscited by 4,231
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  13. 2017Non-local Neural Networkscited by 2,290
  14. 2019Searching for MobileNetV3cited by 1,984
  15. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  16. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  17. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  18. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  19. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  20. 2018Self-Attention with Relative Position Representationscited by 1,112
  21. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  22. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  23. 2020Designing Network Design Spacescited by 954
  24. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  25. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  26. 2020Pre-Trained Image Processing Transformercited by 891
  27. 2021Transformer in Transformercited by 632
  28. 2018Do Better ImageNet Models Transfer Better?cited by 560
  29. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  30. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  31. 2019GhostNet: More Features from Cheap Operationscited by 410
  32. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  33. 2017Residual Attention Network for Image Classificationcited by 385
  34. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  35. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  36. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  37. 2018BAM: Bottleneck Attention Modulecited by 171
  38. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 77; 2023: 68; 2024: 82; 2025: 47; 2026: 23.

310 papers.

Sort:

By area:

  1. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  2. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  3. 2022How Do Vision Transformers Work?cited by 305
  4. 2022Vision Transformer with Deformable Attentioncited by 293
  5. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  6. 2022Visual Attention Networkcited by 210
  7. 2022Patches Are All You Need?cited by 201
  8. 2022Focal Modulation Networkscited by 186
  9. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  10. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  11. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  12. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
38 more
  1. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  2. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  3. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  4. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  5. 2022Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimationcited by 100
  6. 2024MobileNetV4 -- Universal Models for the Mobile Ecosystemcited by 98
  7. 2024Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chipscited by 97
  8. 2022Vision Transformers provably learn spatial structurecited by 92
  9. 2023CompletionFormer: Depth Completion with Convolutions and Vision Transformerscited by 92
  10. 2021Shunted Self-Attention via Multi-Scale Token Aggregationcited by 90
  11. 2023MedViT: A Robust Vision Transformer for Generalized Medical Image Classificationcited by 89
  12. 2022Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesiscited by 85
  13. 2022Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognitioncited by 81
  14. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 81
  15. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  16. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  17. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  18. 2024Fusion-Mamba for Cross-modality Object Detectioncited by 67
  19. 2021Hire-MLP: Vision MLP via Hierarchical Rearrangementcited by 66
  20. 2022Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarioscited by 65
  21. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 65
  22. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  23. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  24. 2024Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Modelcited by 62
  25. 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61
  26. 2024ContiFormer: Continuous-Time Transformer for Irregular Time Series Modelingcited by 61
  27. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  28. 2022Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Localitycited by 57
  29. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 55
  30. 2023Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanismcited by 51
  31. 2024U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformerscited by 51
  32. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  33. 2023Scale-Aware Modulation Meet Transformercited by 49
  34. 2023SpectFormer: Frequency and Attention is what you need in a Vision Transformercited by 48
  35. 2022MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Modelscited by 47
  36. 2023Vision Transformer with Quadrangle Attentioncited by 46
  37. 2022MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Featurescited by 44
  38. 2024A Survey on Transformer Compressioncited by 44

50 most cited of 310 All 310 (JSON)