# Vision Transformers in 2022: An Update on Tiny ImageNet

arXiv: https://arxiv.org/abs/2205.10660 · HTML: https://mlsanity.com/paper/2205.10660/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 21 May 2022.

## Builds on (most cited first, 16 of 16)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [MiniViT: Compressing Vision Transformers with Weight Multiplexing](https://mlsanity.com/paper/2204.07154/) (2022)

## Built on by (most cited first, 13 of 13)

- [When Prompt-based Incremental Learning Does Not Meet Strong Pretraining](https://mlsanity.com/paper/2308.10445/) (2023)
- [Learn to Accumulate Evidence from All Training Samples: Theory and Practice](https://mlsanity.com/paper/2306.11113/) (2023)
- [MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention](https://mlsanity.com/paper/2211.13955/) (2022)
- [Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences](https://mlsanity.com/paper/2210.11794/) (2022)
- [Astroformer: More Data Might not be all you need for Classification](https://mlsanity.com/paper/2304.05350/) (2023)
- [Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge](https://mlsanity.com/paper/2407.12808/) (2024)
- [CNN and ViT Efficiency Study on Tiny ImageNet and DermaMNIST Datasets](https://mlsanity.com/paper/2505.08259/) (2025)
- [A Model Zoo of Vision Transformers](https://mlsanity.com/paper/2504.10231/) (2025)
- [Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains](https://mlsanity.com/paper/2512.22664/) (2025)
- [Generalized Regularized Evidential Deep Learning Models: Theory and Comprehensive Evaluation](https://mlsanity.com/paper/2512.23753/) (2025)
- [Autoguided Online Data Curation for Diffusion Model Training](https://mlsanity.com/paper/2509.15267/) (2025)
- [Batch Normalization Amplifies Memorization and Privacy Risks](https://mlsanity.com/paper/2605.24420/) (2026)
- [Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis](https://mlsanity.com/paper/2605.07888/) (2026)

## Abstract (arXiv)

The recent advances in image transformers have shown impressive results and have largely closed the gap between traditional CNN architectures. The standard procedure is to train on large datasets like ImageNet-21k and then finetune on ImageNet-1k. After finetuning, researches will often consider the transfer learning performance on smaller datasets such as CIFAR-10/100 but have left out Tiny ImageNet. This paper offers an update on vision transformers' performance on Tiny ImageNet. I include Vision Transformer (ViT) , Data Efficient Image Transformer (DeiT), Class Attention in Image Transformer (CaiT), and Swin Transformers. In addition, Swin Transformers beats the current state-of-the-art result with a validation accuracy of 91.35%. Code is available here: https://github.com/ehuynh1106/TinyImageNet-Transformers