mlsanityData to 7 Oct 2026

Vision Transformer Pruning

arXiv 2104.08500 ↗First posted 17 Apr 2021cs.CVcited by 77

Not on the map. Why?

Abstract (arXiv)

Vision transformer has achieved competitive performance on a variety of computer vision applications. However, their storage, run-time memory, and computational demands are hindering the deployment to mobile devices. Here we present a vision transformer pruning approach, which identifies the impacts of dimensions in each layer of transformer and then executes pruning accordingly. By encouraging dimension-wise sparsity in the transformer, important dimensions automatically emerge. A great number of dimensions with small importance scores can be discarded to achieve a high pruning ratio without significantly compromising accuracy. The pipeline for vision transformer pruning is as follows: 1) training with sparsity regularization; 2) pruning dimensions of linear projections; 3) fine-tuning. The reduced parameters and FLOPs ratios of the proposed algorithm are well evaluated and analyzed on ImageNet dataset to demonstrate the effectiveness of our proposed method.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 17 Apr 2021.

Builds on

39 of 48 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
27 more
  1. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2020Pre-Trained Image Processing Transformercited by 891
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2019Contrastive Representation Distillationcited by 734
  10. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  11. 2021Transformer in Transformercited by 632
  12. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  13. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  14. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  15. 2015Deep Learning with Limited Numerical Precisioncited by 368
  16. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  17. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  18. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  19. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  20. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  21. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  22. 2015Convolutional neural networks with low-rank regularizationcited by 102
  23. 2019Star-Transformercited by 94
  24. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
  25. 2020Searching for Low-Bit Weights in Quantized Neural Networkscited by 43
  26. 2019Learning Low-Rank Approximation for CNNscited by 7
  27. 2020An Investigation on Different Underlying Quantization Schemes for Pre-trained Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 21; 2024: 28; 2025: 7; 2026: 10.

77 papers.

Sort:

By area:

  1. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  2. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  4. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  5. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  6. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  7. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  8. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  9. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  10. 2024A Survey on Transformer Compressioncited by 44
  11. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  12. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
38 more
  1. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  2. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  3. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  4. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  5. 2024Isomorphic Pruning for Vision Modelscited by 21
  6. 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
  7. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  8. 2023Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Trainingcited by 19
  9. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  10. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  11. 2024The Need for Speed: Pruning Transformers with One Recipecited by 15
  12. 2024LPViT: Low-Power Semi-structured Pruning for Vision Transformerscited by 15
  13. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  14. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  15. 2024Comprehensive Survey of Model Compression and Speed up for Vision Transformerscited by 14
  16. 2021Multi-Dimensional Model Compression of Vision Transformercited by 13
  17. 2023DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devicescited by 13
  18. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  19. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  20. 2023Robust Transformer with Locality Inductive Bias and Feature Normalizationcited by 12
  21. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  22. 2022The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruningcited by 9
  23. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTscited by 9
  24. 2024UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformercited by 9
  25. 2024Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compressioncited by 8
  26. 2024Dense Vision Transformer Compression with Few Samplescited by 8
  27. 2024Famba-V: Fast Vision Mamba with Cross-Layer Token Fusioncited by 8
  28. 2024Edge Unlearning is Not "on Edge"! An Adaptive Exact Unlearning System on Resource-Constrained Devicescited by 8
  29. 2023Core-Periphery Principle Guided Redesign of Self-Attention in Transformerscited by 7
  30. 2024Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inferencecited by 6
  31. 2022Pruning-aware Sparse Regularization for Network Pruningcited by 5
  32. 2022Vision Transformer Compression with Structured Pruning and Low Rank Approximationcited by 4
  33. 2024Convolutional Initialization for Data-Efficient Vision Transformerscited by 4
  34. 2024Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategycited by 4
  35. 2025MDP: Multidimensional Vision Model Pruning with Latency Constraintcited by 4
  36. 2024Diffusion Model Compression for Image-to-Image Translationcited by 3
  37. 2024Automatic Channel Pruning for Multi-Head Attentioncited by 3
  38. 2022Searching Intrinsic Dimensions of Vision Transformerscited by 2

50 most cited of 77 All 77 (JSON)