mlsanityData to 7 Oct 2026

Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation

arXiv 2308.01045 ↗First posted 2 Aug 2023cs.CVcited by 61

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision transformers have achieved leading performance on various visual tasks yet still suffer from high computational complexity. The situation deteriorates in dense prediction tasks like semantic segmentation, as high-resolution inputs and outputs usually imply more tokens involved in computations. Directly removing the less attentive tokens has been discussed for the image classification task but can not be extended to semantic segmentation since a dense prediction is required for every patch. To this end, this work introduces a Dynamic Token Pruning (DToP) method based on the early exit of tokens for semantic segmentation. Motivated by the coarse-to-fine segmentation process by humans, we naturally split the widely adopted auxiliary-loss-based network architecture into several stages, where each auxiliary block grades every token's difficulty level. We can finalize the prediction of easy tokens in advance without completing the entire forward pass. Moreover, we keep k highest confidence tokens for each semantic category to uphold the representative context information. Thus, computational complexity will change with the difficulty of the input, akin to the way humans do segmentation. Experiments suggest that the proposed DToP architecture reduces on average 20% - 35% of computational cost for current semantic segmentation methods based on plain vision transformers without accuracy degradation.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 35 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

29 of 35 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2016Layer Normalizationcited by 5,579
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
17 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  6. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  7. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  8. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  10. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  11. 2020Context Prior for Scene Segmentationcited by 121
  12. 2018Learning a Discriminative Feature Network for Semantic Segmentationcited by 114
  13. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  14. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  15. 2017Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascadecited by 53
  16. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  17. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 24; 2026: 23.

61 papers.

Sort:

By area:

  1. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  2. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  3. 2024SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentationcited by 31
  4. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  5. 2024Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detectioncited by 18
  6. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  7. 2024SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalizationcited by 17
  8. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  9. 2024Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentationcited by 14
  10. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  11. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  12. 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 9
38 more
  1. 2024Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inferencecited by 6
  2. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  3. 2024Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunablecited by 4
  4. 2025AutoPrune: Each Complexity Deserves a Pruning Policycited by 4
  5. 2024Automatic Channel Pruning for Multi-Head Attentioncited by 3
  6. 2025DUDA: Distilled Unsupervised Domain Adaptation for Lightweight Semantic Segmentationcited by 3
  7. 2025S2AFormer: Strip Self-Attention for Efficient Vision Transformercited by 3
  8. 2025Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purgingcited by 3
  9. 2025Fast SAM2 with Text-Driven Token Pruningcited by 3
  10. 2026Generative Event Pretraining with Foundation Model Alignmentcited by 3
  11. 2026Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruningcited by 3
  12. 2024Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representationscited by 2
  13. 2025When Less is Enough: Adaptive Token Reduction for Efficient Image Representationcited by 2
  14. 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Modelscited by 2
  15. 2026ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extractioncited by 2
  16. 2026Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformerscited by 2
  17. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 2
  18. 2024HCPM: Hierarchical Candidates Pruning for Efficient Detector-Free Matchingcited by 1
  19. 2025Exploring Token-Level Augmentation in Vision Transformer for Semi-Supervised Semantic Segmentationcited by 1
  20. 2025Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruningcited by 1
  21. 2025Two-stage Vision Transformers and Hard Masking offer Robust Object Representationscited by 1
  22. 2025ReGATE: Learning Faster and Better with Fewer Tokens in MLLMscited by 1
  23. 2025Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformerscited by 1
  24. 2026Nonparametric Teaching of Attention Learnerscited by 1
  25. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMscited by 1
  26. 2026Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefillingcited by 1
  27. 2026Na-IRSTD: Enhancing Infrared Small Target Detection via Native-Resolution Feature Selection and Fusioncited by 1
  28. 2025Prompt-based Dynamic Token Pruning for Efficient Segmentation of Medical Imagescited by 0
  29. 2025Deeply Dual Supervised learning for melanoma recognitioncited by 0
  30. 2025Temporal Cluster Assignment for Efficient Real-Time Video Segmentationcited by 0
  31. 2025Edge Detection for Organ Boundaries via Top Down Refinement and SubPixel Upsamplingcited by 0
  32. 2025DualResolution Residual Architecture with Artifact Suppression for Melanocytic Lesion Segmentationcited by 0
  33. 2025VesselRW: Weakly Supervised Subcutaneous Vessel Segmentation via Learned Random Walk Propagationcited by 0
  34. 2025E-BayesSAM: Efficient Bayesian Adaptation of SAM with Self-Optimizing KAN-Based Interpretation for Uncertainty-Aware Ultrasonic Segmentationcited by 0
  35. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutionscited by 0
  36. 2025ClustViT: Clustering-based Token Merging for Semantic Segmentationcited by 0
  37. 2025Collapse of Patches: Ranking Image Patches for Efficient Visual Modelingcited by 0
  38. 2026How to Build Robust, Scalable Models for GSV-Based Indicators in Neighborhood Researchcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation
  2. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  3. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
  4. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers
  5. 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
  6. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  7. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  8. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration