mlsanityData to 7 Oct 2026

MiniViT: Compressing Vision Transformers with Weight Multiplexing

arXiv 2204.07154 ↗First posted 14 Apr 2022cs.CVcited by 88

Not on the map. Why?

Abstract (arXiv)

Vision Transformer (ViT) models have recently drawn much attention in computer vision due to their high model capability. However, ViT models suffer from huge number of parameters, restricting their applicability on devices with limited memory. To alleviate this problem, we propose MiniViT, a new compression framework, which achieves parameter reduction in vision transformers while retaining the same performance. The central idea of MiniViT is to multiplex the weights of consecutive transformer blocks. More specifically, we make the weights shared across layers, while imposing a transformation on the weights to increase diversity. Weight distillation over self-attention is also applied to transfer knowledge from large-scale ViT models to weight-multiplexed compact models. Comprehensive experiments demonstrate the efficacy of MiniViT, showing that it can reduce the size of the pre-trained Swin-B transformer by 48%, while achieving an increase of 1.0% in Top-1 accuracy on ImageNet. Moreover, using a single-layer of parameters, MiniViT is able to compress DeiT-B by 9.7 times from 86M to 9M parameters, without seriously compromising the performance. Finally, we verify the transferability of MiniViT by reporting its performance on downstream benchmarks. Code and models are available at here.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

48 of 66 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Mask R-CNNcited by 7,259
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2016Layer Normalizationcited by 5,579
36 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  15. 2020Designing Network Design Spacescited by 954
  16. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  17. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  18. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  19. 2021Transformer in Transformercited by 632
  20. 2019Deep Equilibrium Modelscited by 603
  21. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  22. 2020Are we done with ImageNet?cited by 329
  23. 2019Revealing the Dark Secrets of BERTcited by 300
  24. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  25. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  26. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  27. 2019Levenshtein Transformercited by 197
  28. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  29. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  30. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  31. 2019PolyTransform: Deep Polygon Transformer for Instance Segmentationcited by 92
  32. 2019A Tensorized Transformer for Language Modelingcited by 85
  33. 2022Unified Visual Transformer Compressioncited by 79
  34. 2021Vision Transformer Pruningcited by 77
  35. 2020Talking-Heads Attentioncited by 57
  36. 2020Grafit: Learning fine-grained image representations with coarse labelscited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 25; 2024: 25; 2025: 13; 2026: 11.

88 papers.

Sort:

By area:

  1. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  2. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  3. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  4. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  5. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  6. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  7. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  8. 2023MixFormerV2: Efficient Fully Transformer Trackingcited by 56
  9. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  10. 2022MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Featurescited by 44
  11. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  12. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
38 more
  1. 2023A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentationcited by 33
  2. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  3. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  4. 2025Similarity-Guided Layer-Adaptive Vision Transformer for UAV Trackingcited by 22
  5. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  6. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
  7. 2024WAVE: Weight Templates for Adaptive Initialization of Variable-sized Modelscited by 16
  8. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  9. 2023Patch-wise Mixed-Precision Quantization of Vision Transformercited by 15
  10. 2022Vision Transformers in 2022: An Update on Tiny ImageNetcited by 13
  11. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  12. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  13. 2023Transformer as Linear Expansion of Learngenecited by 13
  14. 2024From Efficient Multimodal Models to World Models: A Surveycited by 13
  15. 2024KIND: Knowledge Integration and Diversion for Training Decomposable Modelscited by 10
  16. 2024General Compression Framework for Efficient Transformer Object Trackingcited by 10
  17. 2022Efficient High-Resolution Deep Learning: A Surveycited by 9
  18. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTscited by 9
  19. 2024Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Modelscited by 9
  20. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  21. 2025LiT: Delving into a Simple Linear Diffusion Transformer for Image Generationcited by 9
  22. 2022HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compressioncited by 8
  23. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformerscited by 8
  24. 2023Lightweight Full-Convolutional Siamese Trackercited by 8
  25. 2024LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagationcited by 8
  26. 2024Dense Vision Transformer Compression with Few Samplescited by 8
  27. 2022Dynamic Feature Pruning and Consolidation for Occluded Person Re-Identificationcited by 7
  28. 2025Not Every Patch is Needed: Towards a More Efficient and Effective Backbone for Video-based Person Re-identificationcited by 7
  29. 2023Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understandingcited by 6
  30. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  31. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  32. 2022A Generic Shared Attention Mechanism for Various Backbone Neural Networkscited by 5
  33. 2023Towards Efficient Task-Driven Model Reprogramming with Foundation Modelscited by 5
  34. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  35. 2023A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniquescited by 4
  36. 2023Group Multi-View Transformer for 3D Shape Analysis with Spatial Encodingcited by 4
  37. 2024Head-wise Shareable Attention for Large Language Modelscited by 4
  38. 2024Understanding Neural Network Binarization with Forward and Backward Proximal Quantizerscited by 4

50 most cited of 88 All 88 (JSON)