mlsanityData to 14 Oct 2026

Which Tokens to Use? Investigating Token Reduction in Vision Transformers

arXiv 2308.04657 ↗First posted 9 Aug 2023cs.CVcited by 60

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Since the introduction of the Vision Transformer (ViT), researchers have sought to make ViTs more efficient by removing redundant information in the processed tokens. While different methods have been explored to achieve this goal, we still lack understanding of the resulting reduction patterns and how those patterns differ across token reduction methods and datasets. To close this gap, we set out to understand the reduction patterns of 10 different token reduction methods using four image classification datasets. By systematically comparing these methods on the different classification tasks, we find that the Top-K pruning method is a surprisingly strong baseline. Through in-depth analysis of the different methods, we determine that: the reduction patterns are generally not consistent when varying the capacity of the backbone model, the reduction patterns of pruning-based methods significantly differ from fixed radial patterns, and the reduction patterns of pruning-based methods are correlated across classification datasets. Finally we report that the similarity of reduction patterns is a moderate-to-strong proxy for model performance. Project page at https://vap.aau.dk/tokens.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 64 of its 81 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

64 of 81 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning11 Oct
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  4. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
38 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2017Random Erasing Data Augmentationcited by 1,239
  5. 2016Deep Networks with Stochastic Depthcited by 979
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  7. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  8. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  9. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  10. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  12. 2020Asymmetric Loss For Multi-Label Classificationcited by 381
  13. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  14. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  15. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  16. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  17. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  18. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  19. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  20. 2022FlexiViT: One Model for All Patch Sizescited by 152
  21. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  22. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 144
  23. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  24. 2021TransFG: A Transformer Architecture for Fine-grained Recognitioncited by 132
  25. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  26. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 101
  27. 2021The Efficiency Misnomercited by 98
  28. 2019MultiGrain: a unified image embedding for classes and instancescited by 85
  29. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  30. 2022Learning to Merge Tokens in Vision Transformerscited by 65
  31. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  32. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  33. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  34. 2021Vision Transformer with Progressive Samplingcited by 44
  35. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  36. 2021Feature Fusion Vision Transformer for Fine-Grained Visual Categorizationcited by 40
  37. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  38. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 19; 2026: 14.

60 papers.

Sort:

By area:

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  2. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 98
  3. 2024Matryoshka Multimodal Modelscited by 84
  4. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  5. 2024Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identificationcited by 35
  6. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
  7. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  8. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  9. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  10. 2024Subobject-level Image Tokenizationcited by 21
  11. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 19
  12. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
38 more
  1. 2025AVGGT: Rethinking Global Attention for Accelerating VGGTcited by 16
  2. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  3. 2024Frontiers in Intelligent Colonoscopycited by 14
  4. 2024Agglomerative Token Clusteringcited by 13
  5. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  6. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  7. 2024Conditional computation in neural networks: principles and research trendscited by 10
  8. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edgecited by 9
  9. 2024Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Modelcited by 8
  10. 2024freePruner: A Training-free Approach for Large Multimodal Model Accelerationcited by 8
  11. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  12. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Modelscited by 5
  13. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervisioncited by 5
  14. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  15. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  16. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preservedcited by 4
  17. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  18. 2025Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applicationscited by 4
  19. 2024Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformerscited by 3
  20. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AIcited by 3
  21. 2024PO-GUISE+: Pose and object guided transformer token selection for efficient driver action recognitioncited by 2
  22. 2024Token Turing Machines are Efficient Vision Modelscited by 2
  23. 2024TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Modelscited by 2
  24. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retrainingcited by 2
  25. 2025ResidualViT for Efficient Temporally Dense Video Encodingcited by 2
  26. 2025MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classificationcited by 2
  27. 2026From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognitioncited by 2
  28. 2026SAT: Selective Aggregation Transformer for Image Super-Resolutioncited by 2
  29. 2025Training-free Token Reduction for Vision Mambacited by 1
  30. 2025Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformerscited by 1
  31. 2026ASAP: Attention-Shift-Aware Pruning for Efficient LVLM Inferencecited by 1
  32. 2026RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformercited by 1
  33. 2023Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentationcited by 0
  34. 2024Plug-and-Play Performance Estimation for LLM Services without Relying on Labeled Datacited by 0
  35. 2024Training Noise Token Pruningcited by 0
  36. 2025Temporal Cluster Assignment for Efficient Real-Time Video Segmentationcited by 0
  37. 2025Token-Based Detection of Spurious Correlations in Vision Transformerscited by 0
  38. 2025Transformers from Compressed Representationscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient
  2. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
  3. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers
  4. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation
  5. 2023No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
  6. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  7. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  8. 2024Token Pruning using a Lightweight Background Aware Vision Transformer