mlsanityData to 7 Oct 2026

LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference

arXiv 2104.01136 ↗First posted 2 Apr 2021cs.CVcited by 475

Not on the map. Why?

Abstract (arXiv)

We design a family of image classification architectures that optimize the trade-off between accuracy and efficiency in a high-speed regime. Our work exploits recent findings in attention-based architectures, which are competitive on highly parallel processing hardware. We revisit principles from the extensive literature on convolutional neural networks to apply them to transformers, in particular activation maps with decreasing resolutions. We also introduce the attention bias, a new way to integrate positional information in vision transformers. As a result, we propose LeVIT: a hybrid neural network for fast inference image classification. We consider different measures of efficiency on different hardware platforms, so as to best reflect a wide range of application scenarios. Our extensive experiments empirically validate our technical choices and show they are suitable to most architectures. Overall, LeViT significantly outperforms existing convnets and vision transformers with respect to the speed/accuracy tradeoff. For example, at 80% ImageNet top-1 accuracy, LeViT is 5 times faster than EfficientNet on CPU. We release the code at https://github.com/facebookresearch/LeViT

Key facts. Cited by 475 papers in the map (data through 7 Oct 2026). Builds on 57 of its 71 references that are in the map. First posted on arXiv 2 Apr 2021.

Builds on

57 of 71 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2016Densely Connected Convolutional Networkscited by 6,691
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  11. 2017Squeeze-and-Excitation Networkscited by 4,763
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2016Wide Residual Networkscited by 3,057
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  4. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2019Searching for MobileNetV3cited by 1,984
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  13. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  16. 2018Self-Attention with Relative Position Representationscited by 1,112
  17. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  18. 2016Deep Networks with Stochastic Depthcited by 973
  19. 2020Designing Network Design Spacescited by 954
  20. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  21. 2018Image Transformercited by 770
  22. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  23. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  24. 2017Convolutional Sequence to Sequence Learningcited by 550
  25. 2020ResNeSt: Split-Attention Networkscited by 445
  26. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  27. 2019Selective Kernel Networkscited by 401
  28. 2017Residual Attention Network for Image Classificationcited by 385
  29. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  30. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  31. 2019Attention Augmented Convolutional Networkscited by 363
  32. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  33. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  34. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  35. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  36. 2020Are we done with ImageNet?cited by 329
  37. 2020Exploring Self-attention for Image Recognitioncited by 313
  38. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 67; 2022: 112; 2023: 105; 2024: 93; 2025: 78; 2026: 20.

475 papers.

Sort:

By area:

  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2022Token Merging: Your ViT But Fastercited by 995
  3. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  4. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  5. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  6. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 508
  7. 2022DeiT III: Revenge of the ViTcited by 417
  8. 2021Early Convolutions Help Transformers See Bettercited by 395
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  10. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  11. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  12. 2022How Do Vision Transformers Work?cited by 305
38 more
  1. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  2. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  3. 2022Transformers in Medical Imaging: A Surveycited by 245
  4. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  5. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  6. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  7. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  8. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  9. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  10. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  11. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  12. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  13. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  14. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  15. 2022DaViT: Dual Attention Vision Transformerscited by 174
  16. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  17. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  18. 2022Masked World Models for Visual Controlcited by 169
  19. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  20. 2021Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformerscited by 159
  21. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150
  22. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  23. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  24. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  25. 2023Spike-driven Transformercited by 141
  26. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  27. 2021Towards Robust Vision Transformercited by 137
  28. 2023Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networkscited by 129
  29. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  30. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  31. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  32. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  33. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  34. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  35. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  36. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  37. 2021MISSFormer: An Effective Medical Image Segmentation Transformercited by 103
  38. 2021AS-MLP: An Axial Shifted MLP Architecture for Visioncited by 101

50 most cited of 475 All 475 (JSON)