mlsanityData to 14 Oct 2026

When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression

arXiv 2607.02237 ↗First posted 2 Jul 2026cs.CVcited by 0

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision Transformers (ViTs) are strong backbones for semantic segmentation, but their computational cost limits deployment. Recent token compression methods for efficient transformer-based segmentation reduce this cost by decreasing the number of tokens. However, existing evaluations primarily focus on low-to-moderate compression, leaving their behavior under aggressive compression and corrupted inputs unclear. Meanwhile, structural pruning provides an orthogonal route to efficiency by removing redundant components in the ViT architecture, but is rarely compared to token compression under a unified protocol. To bridge this gap, we benchmark representative token compression and structural pruning methods for ViT-based semantic segmentation under matched FLOPs on ADE20K and Cityscapes, together with their common-corruption variants ADE20K-C and Cityscapes-C. Our results reveal a consistent trend on both clean and corrupted inputs: token compression is highly effective at mild reductions but degrades sharply when compression becomes severe, consistent with substantial information loss from overly aggressive token reduction. In contrast, structural pruning exhibits a smoother degradation curve and is more stable at high compression. Motivated by these findings, we study a prune-then-merge pipeline that applies moderate token compression on top of a moderately pruned backbone. At comparable FLOPs, this combined strategy consistently achieves a better accuracy-robustness trade-off at high compression, offering a practical recipe for deployment-oriented ViT segmentation. Code is available at https://github.com/phatnguyencs/vit-seg-compression.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 42 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

34 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers14 Jun
  2. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  3. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformers9 Aug
  4. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation2 Aug
  5. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  6. 2022Token Merging: Your ViT But Faster17 Oct
  7. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec

Other references · 27

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2022A ConvNet for the 2020scited by 3,579
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
15 more
  1. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  2. 2023DepGraph: Towards Any Structural Pruningcited by 203
  3. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  4. 2019What Do Compressed Deep Neural Networks Forget?cited by 171
  5. 2019Benchmarking the Robustness of Semantic Segmentation Modelscited by 111
  6. 2022TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentationcited by 82
  7. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 70
  8. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
  9. 2022Structural Pruning via Latency-Saliency Knapsackcited by 41
  10. 2023Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanismcited by 33
  11. 2023Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformerscited by 22
  12. 2024Isomorphic Pruning for Vision Modelscited by 21
  13. 2024LDReg: Local Dimensionality Regularized Self-Supervised Learningcited by 11
  14. 2025Frequency-Dynamic Attention Modulation for Dense Predictioncited by 10
  15. 2023Dynamic Token-Pass Transformers for Semantic Segmentationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
  2. 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
  3. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs
  4. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  5. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
  6. 2024Token Cropr: Faster ViTs for Quite a Few Tasks
  7. 2026Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
  8. 2025Similarity-Aware Token Pruning: Your VLM but Faster