mlsanityData to 14 Oct 2026

Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers

arXiv 2304.10716 ↗First posted 21 Apr 2023cs.CVcited by 81

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 44 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

37 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  2. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  9. 2021Vision Transformers for Dense Predictioncited by 1,803
  10. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  12. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
23 more
  1. 2021Multiscale Vision Transformerscited by 897
  2. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  3. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  4. 2021Transformer in Transformercited by 633
  5. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  6. 2021TransReID: Transformer-based Object Re-Identificationcited by 435
  7. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  8. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  9. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 286
  10. 2021Global Filter Networks for Image Classificationcited by 260
  11. 2021Voxel Transformer for 3D Object Detectioncited by 247
  12. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  13. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  14. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  15. 20203D Object Detection with Pointformercited by 194
  16. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 183
  17. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  18. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  19. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  20. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 101
  21. 2021Training Vision Transformers for Image Retrievalcited by 90
  22. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  23. 2021Vision Transformer with Progressive Samplingcited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 30; 2025: 25; 2026: 19.

81 papers.

Sort:

By area:

  1. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 126
  2. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  3. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 95
  4. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  5. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  6. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  7. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformerscited by 38
  8. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  9. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  10. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  11. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 35
  12. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 31
38 more
  1. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  2. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  3. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformerscited by 21
  4. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  5. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 19
  6. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  7. 2024MLP Can Be A Good Transformer Learnercited by 17
  8. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  9. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  10. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  11. 2024Agglomerative Token Clusteringcited by 13
  12. 2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
  13. 2023F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesiscited by 10
  14. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformercited by 10
  15. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTscited by 9
  16. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edgecited by 9
  17. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Accelerationcited by 9
  18. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  19. 2025Not Every Patch is Needed: Towards a More Efficient and Effective Backbone for Video-based Person Re-identificationcited by 7
  20. 2024Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuningcited by 6
  21. 2025Dynamic Vision Mambacited by 6
  22. 2025VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Drivingcited by 6
  23. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  24. 2026HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Drivingcited by 5
  25. 2023RingMo-lite: A Remote Sensing Multi-task Lightweight Network with CNN-Transformer Hybrid Frameworkcited by 4
  26. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformercited by 4
  27. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 4
  28. 2024Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancycited by 4
  29. 2024Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunablecited by 4
  30. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  31. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preservedcited by 4
  32. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  33. 2025One-Shot Knowledge Transfer for Scalable Person Re-Identificationcited by 4
  34. 2024Automatic Channel Pruning for Multi-Head Attentioncited by 3
  35. 2025S2AFormer: Strip Self-Attention for Efficient Vision Transformercited by 3
  36. 2025Diversity-Guided MLP Reduction for Efficient Large Vision Transformerscited by 3
  37. 2025Frequency-Aware Token Reduction for Efficient Vision Transformercited by 3
  38. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 3

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
  2. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers
  3. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient
  4. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformers
  5. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
  6. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  7. 2022Unified Visual Transformer Compression
  8. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation