mlsanityData to 7 Oct 2026

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

arXiv 2103.10697 ↗First posted 19 Mar 2021cs.CV cs.LG stat.MLcited by 484

Not on the map. Why?

Abstract (arXiv)

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ``soft" convolutional inductive bias. We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analysing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.

Key facts. Cited by 484 papers in the map (data through 7 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 19 Mar 2021.

Builds on

33 of 57 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2017Squeeze-and-Excitation Networkscited by 4,763
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
21 more
  1. 2020Designing Network Design Spacescited by 954
  2. 2020Object-Centric Learning with Slot Attentioncited by 865
  3. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  4. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  5. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  7. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  8. 2019Attention Augmented Convolutional Networkscited by 363
  9. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  10. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  11. 2015LSTM: A Search Space Odysseycited by 315
  12. 2017Relation Networks for Object Detectioncited by 305
  13. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  14. 2019Adaptive Attention Span in Transformerscited by 148
  15. 2018Gather-Excite: Exploiting Feature Context in Convolutional Neural Networkscited by 138
  16. 2018A2-Nets: Double Attention Networkscited by 136
  17. 2020Towards Learning Convolutions from Scratchcited by 46
  18. 2020Transferring Inductive Biases through Knowledge Distillationcited by 41
  19. 2020Revisiting Spatial Invariance with Low-Rank Local Connectivitycited by 29
  20. 2019Finding the Needle in the Haystack with Convolutions: on the benefits of architectural biascited by 19
  21. 2016Homotopy Analysis for Tensor PCAcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 79; 2022: 134; 2023: 101; 2024: 79; 2025: 55; 2026: 36.

484 papers.

Sort:

By area:

  1. 2022A ConvNet for the 2020scited by 3,545
  2. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  3. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  4. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  5. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  6. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  7. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  8. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  9. 2022DeiT III: Revenge of the ViTcited by 417
  10. 2021Early Convolutions Help Transformers See Bettercited by 395
  11. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  12. 2021XCiT: Cross-Covariance Image Transformerscited by 335
38 more
  1. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  2. 2022How Do Vision Transformers Work?cited by 305
  3. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  4. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  5. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  6. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  7. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  8. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  9. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  10. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  11. 2023Class-Incremental Learning: A Surveycited by 209
  12. 2022Patches Are All You Need?cited by 201
  13. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  14. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  15. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  16. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  17. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  18. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  19. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  20. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  21. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  22. 2021Towards Robust Vision Transformercited by 137
  23. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  24. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  25. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  26. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  27. 2024Rethinking Inductive Biases for Surface Normal Estimationcited by 122
  28. 2022EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applicationscited by 121
  29. 2021A Survey of Visual Transformerscited by 117
  30. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  31. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  32. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
  33. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  34. 2022HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutionscited by 103
  35. 2023Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolationcited by 97
  36. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  37. 2022Approximately Equivariant Networks for Imperfectly Symmetric Dynamicscited by 95
  38. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94

50 most cited of 484 All 484 (JSON)