mlsanityData to 13 Oct 2026

Patch Slimming for Efficient Vision Transformers

arXiv 2106.02852 ↗First posted 5 Jun 2021cs.CV cs.LGcited by 132

Not on the map. Why?

Cited by 25 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a top-down paradigm. We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers. For each layer, the impact of a patch on the final output feature is approximated and patches with less impact will be removed. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers without affecting their performances. For example, over 45% FLOPs of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the ImageNet dataset.

Key facts. Cited by 132 papers in the map (data through 13 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

32 of 50 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,315
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,017
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,961
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,138
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2020Pre-Trained Image Processing Transformercited by 892
20 more
  1. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 849
  3. 2019Are Sixteen Heads Really Better than One?cited by 807
  4. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 701
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 675
  6. 2021Transformer in Transformercited by 633
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  8. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  9. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  10. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  11. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  12. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  13. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 93
  14. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  15. 2015Fast ConvNets Using Group-wise Brain Damagecited by 82
  16. 2018Channel Gating Neural Networkscited by 80
  17. 2021Manifold Regularized Dynamic Network Pruningcited by 43
  18. 2020Training Binary Neural Networks through Learning with Noisy Supervisioncited by 39
  19. 2021Locally Free Weight Sharing for Network Width Searchcited by 27
  20. 2021BCNet: Searching for Network Width with Bilaterally Coupled Networkcited by 23

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 16; 2022: 20; 2023: 38; 2024: 27; 2025: 20; 2026: 11.

132 papers.

Sort:

By area:

  1. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  2. 2021Post-Training Quantization for Vision Transformercited by 278
  3. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  4. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 177
  5. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  6. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 166
  7. 2022FlexiViT: One Model for All Patch Sizescited by 152
  8. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 137
  9. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  10. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  11. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 94
  12. 2022TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentationcited by 82
38 more
  1. 2022Unified Visual Transformer Compressioncited by 79
  2. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  3. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  4. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 77
  5. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  6. 2021Hire-MLP: Vision MLP via Hierarchical Rearrangementcited by 66
  7. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  8. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  9. 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61
  10. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillationcited by 60
  11. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  12. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  13. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 56
  14. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  15. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  16. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  17. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  18. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  19. 2024A Survey on Transformer Compressioncited by 46
  20. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  21. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  22. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  23. 2023Less is More: Focus Attention for Efficient DETRcited by 38
  24. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  25. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  26. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  27. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  28. 2023LGViT: Dynamic Early Exiting for Accelerating Vision Transformercited by 30
  29. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  30. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  31. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  32. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 28
  33. 2021Self-slimmed Vision Transformercited by 27
  34. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 24
  35. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  36. 2025Similarity-Guided Layer-Adaptive Vision Transformer for UAV Trackingcited by 23
  37. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  38. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20

50 most cited of 132 All 132 (JSON)