mlsanityData to 7 Oct 2026

Revisiting Token Pruning for Object Detection and Instance Segmentation

arXiv 2306.07050 ↗First posted 12 Jun 2023cs.CVcited by 50

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision Transformers (ViTs) have shown impressive performance in computer vision, but their high computational cost, quadratic in the number of tokens, limits their adoption in computation-constrained applications. However, this large number of tokens may not be necessary, as not all tokens are equally important. In this paper, we investigate token pruning to accelerate inference for object detection and instance segmentation, extending prior works from image classification. Through extensive experiments, we offer four insights for dense tasks: (i) tokens should not be completely pruned and discarded, but rather preserved in the feature maps for later use. (ii) reactivating previously pruned tokens can further enhance model performance. (iii) a dynamic pruning rate based on images is better than a fixed pruning rate. (iv) a lightweight, 2-layer MLP can effectively prune tokens, achieving accuracy comparable with complex gating networks with a simpler design. We assess the effects of these design decisions on the COCO dataset and introduce an approach that incorporates these findings, showing a reduction in performance decline from ~1.5 mAP to ~0.3 mAP in both boxes and masks, compared to existing token pruning methods. In relation to the dense counterpart that utilizes all tokens, our method realizes an increase in inference speed, achieving up to 34% faster performance for the entire network and 46% for the backbone.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 70 of its 79 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

70 of 79 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
38 more
  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2020Point Transformercited by 1,462
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  13. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  14. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  15. 2021Transformers in Vision: A Surveycited by 1,009
  16. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  17. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  18. 2019Are Sixteen Heads Really Better than One?cited by 803
  19. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  20. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  21. 2020A Survey on Visual Transformercited by 653
  22. 2021Transformer in Transformercited by 632
  23. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  24. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  25. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  26. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  27. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  28. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  29. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  30. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  31. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  32. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  33. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 284
  34. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
  35. 2020Sparse Sinkhorn Attentioncited by 250
  36. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  37. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  38. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 22; 2026: 15.

50 papers.

Sort:

By area:

  1. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 40
  2. 2024MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Maskscited by 39
  3. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  4. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  5. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  6. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  7. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Modelscited by 15
  8. 2024Agglomerative Token Clusteringcited by 13
  9. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  10. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  11. 2024Conditional computation in neural networks: principles and research trendscited by 10
  12. 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 9
38 more
  1. 2024Principles of Visual Tokens for Efficient Video Understandingcited by 7
  2. 2024Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inferencecited by 6
  3. 2025Dynamic Vision Mambacited by 6
  4. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  5. 2025MutualForce: Mutual-Aware Enhancement for 4D Radar-LiDAR 3D Object Detectioncited by 5
  6. 2024Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunablecited by 4
  7. 2024Token Pruning using a Lightweight Background Aware Vision Transformercited by 3
  8. 2025A Hidden Stumbling Block in Generalized Category Discovery: Distracted Attentioncited by 3
  9. 2025Fast SAM2 with Text-Driven Token Pruningcited by 3
  10. 2026Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Groundingcited by 3
  11. 2026Generative Event Pretraining with Foundation Model Alignmentcited by 3
  12. 2025B-FPGM: Lightweight Face Detection via Bayesian-Optimized Soft FPGM Pruningcited by 2
  13. 2025Accelerate 3D Object Detection Models via Zero-Shot Attention Key Pruningcited by 2
  14. 2025When Less is Enough: Adaptive Token Reduction for Efficient Image Representationcited by 2
  15. 2025MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspectivecited by 2
  16. 2026Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Modelscited by 2
  17. 2026Temporal Aware Pruning for Efficient Diffusion-based Video Generationcited by 2
  18. 2025Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformerscited by 1
  19. 2025Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detectioncited by 1
  20. 2025I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentationcited by 1
  21. 2026Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectorscited by 1
  22. 2026An AI4AI Framework for Visual Token Pruningcited by 1
  23. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  24. 2025Detection Transformers Under the Knife: A Neuroscience-Inspired Approach to Ablationscited by 0
  25. 2025Temporal Cluster Assignment for Efficient Real-Time Video Segmentationcited by 0
  26. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutionscited by 0
  27. 2025CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbonescited by 0
  28. 2025PSTTS: A Plug-and-Play Token Selector for Efficient Event-based Spatio-temporal Representation Learningcited by 0
  29. 2025MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mambacited by 0
  30. 2026Reg4Pru: Regularisation Through Random Token Routing for Token Pruningcited by 0
  31. 2026NeuroCanvas: VLLM-Powered Robust Seizure Detection by Reformulating Multichannel EEG as Imagecited by 0
  32. 2026Energy-Aware Imitation Learning for Steering Prediction Using Events and Framescited by 0
  33. 2026Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reductioncited by 0
  34. 2026Certainty Is Redundant: Token Sparsification for Efficient Camouflaged Object Detection with Vision Foundation Modelscited by 0
  35. 2026SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detectioncited by 0
  36. 2026Keep the Needle, Prune the Haystack: Defect-Preserving Token Pruning for Efficient Zero-Shot Anomaly Detectioncited by 0
  37. 2026Putting Registers to Work: Task Registers for Token Pruning in Vision Transformerscited by 0
  38. 2026TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  2. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  3. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation
  4. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  5. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers
  6. 2026RS3-Prune: Read-Sparse, Store-Sparse Token Pruning for Video Object Segmentation
  7. 2023No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
  8. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers