mlsanityData to 7 Oct 2026

Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

arXiv 2202.07800 ↗First posted 16 Feb 2022cs.CV cs.LGcited by 359

Not on the map. Why?

Cited by 115 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at https://github.com/youweiliang/evit

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). Builds on 51 of its 58 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

51 of 58 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  9. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  12. 2021Going deeper with Image Transformerscited by 759
  13. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  14. 2021Transformer in Transformercited by 632
  15. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  16. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  17. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  18. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  19. 2021Intriguing Properties of Vision Transformerscited by 446
  20. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  21. 2021Dynamic Neural Networks: A Surveycited by 405
  22. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  23. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  24. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  25. 2018Video Action Transformer Networkcited by 316
  26. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  27. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  28. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  29. 2019Local Relation Networks for Image Recognitioncited by 210
  30. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  31. 2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
  32. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  33. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  34. 2021Visformer: The Vision-friendly Transformercited by 137
  35. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  36. 2020Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inferencecited by 45
  37. 2021Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learningcited by 34
  38. 2019Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inferencecited by 31

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 26; 2023: 48; 2024: 75; 2025: 100; 2026: 110.

359 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  4. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  5. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  7. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 193
  8. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  9. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  10. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  11. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  12. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
38 more
  1. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  2. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  3. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  4. 2022Dilated Neighborhood Attention Transformercited by 64
  5. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  6. 2024Dynamic Diffusion Transformercited by 60
  7. 2023Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classificationcited by 54
  8. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  9. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  10. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  11. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  12. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  13. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  14. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  15. 2024A Survey on Transformer Compressioncited by 44
  16. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  17. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  18. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  19. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  20. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  21. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39
  22. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  23. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 37
  24. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  25. 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graphcited by 35
  26. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  27. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  28. 2023OcTr: Octree-based Transformer for 3D Object Detectioncited by 32
  29. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  30. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  31. 2024Breaking the Low-Rank Dilemma of Linear Attentioncited by 31
  32. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  33. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  34. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  35. 2024Token Caching for Diffusion Transformer Accelerationcited by 29
  36. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  37. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  38. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28

50 most cited of 359 All 359 (JSON)