# Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets

arXiv: https://arxiv.org/abs/2210.05958 · HTML: https://mlsanity.com/paper/2210.05958/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 91 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 76)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)

## Built on by (most cited first, 29 of 29)

- [Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detection](https://mlsanity.com/paper/2406.17376/) (2024)
- [Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning](https://mlsanity.com/paper/2212.14284/) (2022)
- [Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets](https://mlsanity.com/paper/2407.19394/) (2024)
- [Equipping Computational Pathology Systems with Artifact Processing Pipelines: A Showcase for Computation and Performance Trade-offs](https://mlsanity.com/paper/2403.07743/) (2024)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC](https://mlsanity.com/paper/2312.05705/) (2023)
- [Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesis](https://mlsanity.com/paper/2406.18967/) (2024)
- [GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasets](https://mlsanity.com/paper/2408.14131/) (2024)
- [Multi-Flow: Multi-View-Enriched Normalizing Flows for Industrial Anomaly Detection](https://mlsanity.com/paper/2504.03306/) (2025)
- [GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples](https://mlsanity.com/paper/2305.07931/) (2023)
- [Probing the Efficacy of Federated Parameter-Efficient Fine-Tuning of Vision Transformers for Medical Image Classification](https://mlsanity.com/paper/2407.11573/) (2024)
- [Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge](https://mlsanity.com/paper/2510.19266/) (2025)
- [Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models](https://mlsanity.com/paper/2502.17951/) (2025)
- [HGFormer: Topology-Aware Vision Transformer with HyperGraph Learning](https://mlsanity.com/paper/2504.02440/) (2025)
- [The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach](https://mlsanity.com/paper/2409.17370/) (2024)
- [TCNFormer: Temporal Convolutional Network Former for Short-Term Wind Speed Forecasting](https://mlsanity.com/paper/2408.15737/) (2024)
- [Adaptive Voxel-Weighted Loss Using L1 Norms in Deep Neural Networks for Detection and Segmentation of Prostate Cancer Lesions in PET/CT Images](https://mlsanity.com/paper/2502.02756/) (2025)
- [DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation](https://mlsanity.com/paper/2604.12411/) (2026)
- [IndoHerb: Indonesia Medicinal Plants Recognition using Transfer Learning and Deep Learning](https://mlsanity.com/paper/2308.01604/) (2023)
- [GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets](https://mlsanity.com/paper/2404.04924/) (2024)
- [Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction](https://mlsanity.com/paper/2405.11823/) (2024)
- [Morphological Detection and Classification of Microplastics and Nanoplastics Emerged from Consumer Products by Deep Learning](https://mlsanity.com/paper/2409.13688/) (2024)
- [SiNGER: A Clearer Voice Distills Vision Transformers Further](https://mlsanity.com/paper/2509.20986/) (2025)
- [Shaken or Stirred? An Analysis of MetaFormer's Token Mixing for Medical Imaging](https://mlsanity.com/paper/2510.05971/) (2025)
- [DINOv3-Guided Cross Fusion Framework for Semantic-aware CT generation from MRI and CBCT](https://mlsanity.com/paper/2511.12098/) (2025)
- [Weierstrass Positional Encoding for Vision Transformers](https://mlsanity.com/paper/2605.23719/) (2026)
- [Beyond Accuracy: Evaluating Efficiency, Robustness and Explainability in Deep Learning for Malaria Diagnosis](https://mlsanity.com/paper/2605.30734/) (2026)
- [Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers](https://mlsanity.com/paper/2606.14757/) (2026)
- [Automated carotid artery atherosclerotic inflammation segmentation in PET using MSR-Net](https://mlsanity.com/paper/2610.15245/) (2026)

## Abstract (arXiv)

There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new "head token" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.