mlsanityData to 7 Oct 2026

TinyViT: Fast Pretraining Distillation for Small Vision Transformers

arXiv 2207.10666 ↗First posted 21 Jul 2022cs.CVcited by 232

Not on the map. Why?

Abstract (arXiv)

Vision transformer (ViT) recently has drawn great attention in computer vision due to its remarkable model capability. However, most prevailing ViT models suffer from huge number of parameters, restricting their applicability on devices with limited resources. To alleviate this issue, we propose TinyViT, a new family of tiny and efficient small vision transformers pretrained on large-scale datasets with our proposed fast distillation framework. The central idea is to transfer knowledge from large pretrained models to small ones, while enabling small models to get the dividends of massive pretraining data. More specifically, we apply distillation during pretraining for knowledge transfer. The logits of large teacher models are sparsified and stored in disk in advance to save the memory cost and computation overheads. The tiny student transformers are automatically scaled down from a large pretrained model with computation and parameter constraints. Comprehensive experiments demonstrate the efficacy of TinyViT. It achieves a top-1 accuracy of 84.8% on ImageNet-1k with only 21M parameters, being comparable to Swin-B pretrained on ImageNet-21k while using 4.2 times fewer parameters. Moreover, increasing image resolutions, TinyViT can reach 86.5% accuracy, being slightly better than Swin-L while using only 11% parameters. Last but not the least, we demonstrate a good transfer ability of TinyViT on various downstream tasks. Code and models are available at https://github.com/microsoft/Cream/tree/main/TinyViT.

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). Builds on 68 of its 84 references that are in the map. First posted on arXiv 21 Jul 2022.

Builds on

68 of 84 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2016Layer Normalizationcited by 5,579
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  13. 2019Searching for MobileNetV3cited by 1,984
  14. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2020Knowledge Distillation: A Surveycited by 1,578
  17. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  18. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  19. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  20. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  21. 2017Random Erasing Data Augmentationcited by 1,235
  22. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  23. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  24. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  25. 2021Scaling Vision Transformerscited by 993
  26. 2016Deep Networks with Stochastic Depthcited by 973
  27. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  28. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  31. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  32. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  33. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  34. 2021ImageNet-21K Pretraining for the Massescited by 607
  35. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  36. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  37. 2021Early Convolutions Help Transformers See Bettercited by 395
  38. 2020Are we done with ImageNet?cited by 329

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 44; 2024: 56; 2025: 86; 2026: 42.

232 papers.

Sort:

By area:

  1. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  2. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  3. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  4. 2024Logit Standardization in Knowledge Distillationcited by 101
  5. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  6. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  7. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  8. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  9. 2024A Survey on Transformer Compressioncited by 44
  10. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  11. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  12. 2023EdgeSAM: Prompt-In-the-Loop Distillation for SAMcited by 40
38 more
  1. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  2. 2023TinySAM: Pushing the Envelope for Efficient Segment Anything Modelcited by 33
  3. 2023Robustness of SAM: Segment Anything Under Corruptions and Beyondcited by 30
  4. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  5. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  6. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  7. 2023Learned Thresholds Token Merging and Pruning for Vision Transformerscited by 22
  8. 2023DAT++: Spatially Dynamic Vision Transformer with Deformable Attentioncited by 21
  9. 2025Spiking Vision Transformer with Saccadic Attentioncited by 21
  10. 2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
  11. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  12. 2023Generic-to-Specific Distillation of Masked Autoencoderscited by 18
  13. 2024Learning-based Multi-View Stereo: A Surveycited by 18
  14. 2023RepViT-SAM: Towards Real-Time Segmenting Anythingcited by 17
  15. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  16. 2022Accumulated Trivial Attention Matters in Vision Transformers on Small Datasetscited by 15
  17. 2024BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Modelcited by 15
  18. 2024From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approachcited by 15
  19. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  20. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  21. 2023Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairmentscited by 13
  22. 2023Transformer as Linear Expansion of Learngenecited by 13
  23. 2023Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discoverycited by 13
  24. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
  25. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 13
  26. 2023SPPNet: A Single-Point Prompt Network for Nuclei Image Segmentationcited by 12
  27. 2023nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performancecited by 12
  28. 2024DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTscited by 12
  29. 2025Quantized Spike-driven Transformercited by 12
  30. 2024Towards Natural Machine Unlearningcited by 11
  31. 2024De-LightSAM: Modality-Decoupled Lightweight SAM for Generalizable Medical Segmentationcited by 11
  32. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  33. 2024PitVis-2023 Challenge: Workflow Recognition in videos of Endoscopic Pituitary Surgerycited by 10
  34. 2026AIBuildAI: An AI Agent for Automatically Building AI Modelscited by 10
  35. 2024PosSAM: Panoptic Open-vocabulary Segment Anythingcited by 9
  36. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  37. 2023Lightweight Full-Convolutional Siamese Trackercited by 8
  38. 2024LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagationcited by 8

50 most cited of 232 All 232 (JSON)