mlsanityData to 7 Oct 2026

Learned Thresholds Token Merging and Pruning for Vision Transformers

arXiv 2307.10780 ↗First posted 20 Jul 2023cs.CVcited by 22

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision transformers have demonstrated remarkable success in a wide range of computer vision tasks over the last years. However, their high computational costs remain a significant barrier to their practical deployment. In particular, the complexity of transformer models is quadratic with respect to the number of input tokens. Therefore techniques that reduce the number of input tokens that need to be processed have been proposed. This paper introduces Learned Thresholds token Merging and Pruning (LTMP), a novel approach that leverages the strengths of both token merging and token pruning. LTMP uses learned threshold masking modules that dynamically determine which tokens to merge and which to prune. We demonstrate our approach with extensive experiments on vision transformers on the ImageNet classification task. Our results demonstrate that LTMP achieves state-of-the-art accuracy across reduction rates while requiring only a single fine-tuning epoch, which is an order of magnitude faster than previous methods. Code is available at https://github.com/Mxbonn/ltmp .

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 30 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

21 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention28 Sep
  3. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug
  4. 2021Learned Token Pruning for Transformers2 Jul

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  10. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  11. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  12. 2021Post-Training Quantization for Vision Transformercited by 274
5 more
  1. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  2. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  3. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  4. 2020SoftSort: A Continuous Relaxation for the argsort Operatorcited by 66
  5. 2022Hardware-aware mobile building block evaluation for computer visioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  2. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  3. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  4. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  5. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  6. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  7. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  8. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  9. 2024Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edgecited by 8
  10. 2024Efficient Time Series Processing for Transformers and State-Space Models through Token Mergingcited by 7
  11. 2024Efficient Visual Transformer by Learnable Token Mergingcited by 7
  12. 2025Faster Parameter-Efficient Tuning with Token Redundancy Reductioncited by 4
10 more
  1. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  2. 2024Visual-Word Tokenizer: Beyond Fixed Sets of Tokens in Vision Transformerscited by 2
  3. 2026Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformerscited by 2
  4. 2025P2HCT: Plug-and-Play Hierarchical C2F Transformer for Multi-Scale Feature Fusioncited by 1
  5. 2025Compact Vision Transformer by Reduction of Kernel Complexitycited by 0
  6. 2025Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutionscited by 0
  7. 2026Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signalscited by 0
  8. 2026Make Your LVLM KV Cache More Lightweightcited by 0
  9. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TMcited by 0
  10. 2026Later Is Better: Token Reduction for ViTs Under Distribution Shiftcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  2. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  3. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
  4. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  5. 2023Revisiting Token Pruning for Object Detection and Instance Segmentation
  6. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  7. 2025Efficient Token Compression for Vision Transformer with Spatial Information Preserved
  8. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficient