mlsanityData to 7 Oct 2026

Token Cropr: Faster ViTs for Quite a Few Tasks

arXiv 2412.00965 ↗First posted 1 Dec 2024cs.CV cs.LGcited by 18

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

The adoption of Vision Transformers (ViTs) in resource-constrained applications necessitates improvements in inference throughput. To this end several token pruning and merging approaches have been proposed that improve efficiency by successively reducing the number of tokens. However, it remains an open problem to design a token reduction method that is fast, maintains high performance, and is applicable to various vision tasks. In this work, we present a token pruner that uses auxiliary prediction heads that learn to select tokens end-to-end based on task relevance. These auxiliary heads can be removed after training, leading to throughput close to that of a random pruner. We evaluate our method on image classification, semantic segmentation, object detection, and instance segmentation, and show speedups of 1.5 to 4x with small drops in performance. As a best case, on the ADE20k semantic segmentation benchmark, we observe a 2x speedup relative to the no-pruning baseline, with a negligible performance penalty of 0.1 median mIoU across 5 seeds.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 64 of its 81 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

64 of 81 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  2. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation6 Nov
  3. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers3 Oct
  4. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformers9 Aug
  5. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation2 Aug
  6. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers20 Jul
  7. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation12 Jun
  8. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers21 Apr
  9. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers15 Nov
  10. 2022Token Merging: Your ViT But Faster17 Oct
  11. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning11 Oct
  12. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
2 more
  1. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug
  2. 2021Learned Token Pruning for Transformers2 Jul

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2017Mask R-CNNcited by 7,259
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2021Scaling Vision Transformerscited by 993
  8. 2016Deep Networks with Stochastic Depthcited by 973
  9. 2020Designing Network Design Spacescited by 954
  10. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  12. 2020Quantifying Attention Flow in Transformerscited by 759
  13. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  14. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  15. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  16. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  17. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  18. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  19. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  20. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  21. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  22. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  23. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  24. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  25. 2019Mask Scoring R-CNNcited by 265
  26. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  27. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  28. 2022DaViT: Dual Attention Vision Transformerscited by 174
  29. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  30. 2022FlexiViT: One Model for All Patch Sizescited by 151
  31. 2022MetaFormer Baselines for Visioncited by 131
  32. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 99
  33. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  34. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  35. 2021SOLO: A Simple Framework for Instance Segmentationcited by 32
  36. 2021Self-slimmed Vision Transformercited by 27
  37. 2022Iterative Patch Selection for High-Resolution Image Recognitioncited by 13
  38. 2023Dynamic Token-Pass Transformers for Semantic Segmentationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  2. 2025LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervisioncited by 5
  3. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AIcited by 3
  4. 2025A Hidden Stumbling Block in Generalized Category Discovery: Distracted Attentioncited by 3
  5. 2026MPM: Mutual Pair Merging for Efficient Vision Transformerscited by 2
  6. 2026Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectorscited by 1
  7. 2026Baseline-Relative Counterfactual Refinement for Bit-Aware Visual Token Communicationcited by 1
  8. 2026Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistencycited by 1
  9. 2026R'enyi Attention Entropy for Patch Pruningcited by 0
  10. 2026InterPruner: Interactive Structured Pruning via Taylor-Implicit Criterion and Language-Prior Modulator for Multimodal Object Detectioncited by 0
  11. 2026Putting Registers to Work: Task Registers for Token Pruning in Vision Transformerscited by 0
  12. 2026How Merge-Tolerant Are Vision Transformers for Wheat Phenotyping?cited by 0
6 more
  1. 2026TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Modelscited by 0
  2. 2026LookThere! Sparse Vision by Reinforced Selectioncited by 0
  3. 2026Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Trackingcited by 0
  4. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TMcited by 0
  5. 2026SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Modelscited by 0
  6. 2026Task-Induced Riemannian Metrics for Vision Transformer Feature Spacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  2. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
  3. 2024Token Pruning using a Lightweight Background Aware Vision Transformer
  4. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  5. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
  6. 2026When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression
  7. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMs
  8. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient