mlsanityData to 7 Oct 2026

Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers

arXiv 2211.11315 ↗First posted 21 Nov 2022cs.CVcited by 47

Not on the map. Why?

Cited by 21 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision transformers have achieved significant improvements on various vision tasks but their quadratic interactions between tokens significantly reduce computational efficiency. Many pruning methods have been proposed to remove redundant tokens for efficient vision transformers recently. However, existing studies mainly focus on the token importance to preserve local attentive tokens but completely ignore the global token diversity. In this paper, we emphasize the cruciality of diverse global semantics and propose an efficient token decoupling and merging method that can jointly consider the token importance and diversity for token pruning. According to the class token attention, we decouple the attentive and inattentive tokens. In addition to preserving the most discriminative local tokens, we merge similar inattentive tokens and match homogeneous attentive tokens to maximize the token diversity. Despite its simplicity, our method obtains a promising trade-off between model complexity and classification accuracy. On DeiT-S, our method reduces the FLOPs by 35% with only a 0.2% accuracy drop. Notably, benefiting from maintaining the token diversity, our method can even improve the accuracy of DeiT-T by 0.1% after reducing its FLOPs by 40%.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

35 of 44 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  9. 2020Designing Network Design Spacescited by 954
  10. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  11. 2021Multiscale Vision Transformerscited by 896
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
23 more
  1. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  2. 2021Transformer in Transformercited by 632
  3. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  4. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  5. 2019GhostNet: More Features from Cheap Operationscited by 410
  6. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  7. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  8. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  9. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  10. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  11. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  12. 2022Vision GNN: An Image is Worth Graph of Nodescited by 185
  13. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  14. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  15. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  16. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  17. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  18. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  19. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  20. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  21. 2021Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformerscited by 109
  22. 2021Vision Transformers with Patch Diversificationcited by 45
  23. 2021Augmented Shortcuts for Vision Transformerscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 19; 2025: 16; 2026: 9.

47 papers.

Sort:

By area:

  1. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  2. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39
  3. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  4. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  5. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  6. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  7. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  8. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  9. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  10. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  11. 2024Agglomerative Token Clusteringcited by 13
  12. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
35 more
  1. 2024Leveraging Temporal Contextualization for Video Action Recognitioncited by 9
  2. 2024Make Your ViT-based Multi-view 3D Detectors Faster via Token Compressioncited by 9
  3. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Accelerationcited by 9
  4. 2026AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Modelscited by 7
  5. 2024Data-independent Module-aware Pruning for Hierarchical Vision Transformerscited by 6
  6. 2024SCE-MAE: Selective Correspondence Enhancement with Masked Autoencoder for Self-Supervised Landmark Estimationcited by 6
  7. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
  8. 2023Multi-scale Efficient Graph-Transformer for Whole Slide Image Classificationcited by 5
  9. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  10. 2025Representation Shift: Unifying Token Compression with FlashAttentioncited by 5
  11. 2025H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformerscited by 5
  12. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  13. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  14. 2025Lossless Token Merging Even Without Fine-Tuning in Vision Transformerscited by 4
  15. 2026DocPrune:Efficient Document Question Answering via Background, Question, and Comprehension-aware Token Pruningcited by 4
  16. 2025Diversity-Guided MLP Reduction for Efficient Large Vision Transformerscited by 3
  17. 2025When Less is Enough: Adaptive Token Reduction for Efficient Image Representationcited by 2
  18. 2026Adaptive MLP Pruning for Large Vision Transformerscited by 2
  19. 2026ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extractioncited by 2
  20. 2024Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Modelscited by 1
  21. 2025TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularizationcited by 1
  22. 2025FTCFormer: Fuzzy Token Clustering Transformer for Image Classificationcited by 1
  23. 2024Arena: A Patch-of-Interest ViT Inference Acceleration System for Edge-Assisted Video Analyticscited by 0
  24. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognitioncited by 0
  25. 2025Dual-frequency Selected Knowledge Distillation with Statistical-based Sample Rectification for PolSAR Image Classificationcited by 0
  26. 2025Block-based Symmetric Pruning and Fusion for Efficient Vision Transformerscited by 0
  27. 2025Temporal Cluster Assignment for Efficient Real-Time Video Segmentationcited by 0
  28. 2025Towards Efficient Vision State Space Models via Token Mergingcited by 0
  29. 2025Token-Based Detection of Spurious Correlations in Vision Transformerscited by 0
  30. 2025CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbonescited by 0
  31. 2026ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformerscited by 0
  32. 2026SAGE: Training-Free Semantic Evidence Composition for Edge-Cloud Inference under Hard Uplink Budgetscited by 0
  33. 2026DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservationcited by 0
  34. 2026REDI: Corpus Aware Patch Ranking for DINOv3 Token Reductioncited by 0
  35. 2026Putting Registers to Work: Task Registers for Token Pruning in Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026R'enyi Entropy: A New Token Pruning Metric for Vision Transformers
  2. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  3. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  4. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  5. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  6. 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
  7. 2026ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning
  8. 2024Exploring Token Pruning in Vision State Space Models