mlsanityData to 7 Oct 2026

Training-Free Acceleration of ViTs with Delayed Spatial Merging

arXiv 2303.02331 ↗First posted 4 Mar 2023cs.CV cs.AI cs.LGcited by 4

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Token merging has emerged as a new paradigm that can accelerate the inference of Vision Transformers (ViTs) without any retraining or fine-tuning. To push the frontier of training-free acceleration in ViTs, we improve token merging by adding the perspectives of 1) activation outliers and 2) hierarchical representations. Through a careful analysis of the attention behavior in ViTs, we characterize a delayed onset of the convergent attention phenomenon, which makes token merging undesirable in the bottom blocks of ViTs. Moreover, we augment token merging with a hierarchical processing scheme to capture multi-scale redundancy between visual tokens. Combining these two insights, we build a unified inference framework called DSM: Delayed Spatial Merging. We extensively evaluate DSM on various ViT model scales (Tiny to Huge) and tasks (ImageNet-1k and transfer learning), achieving up to 1.8× FLOP reduction and 1.6× throughput speedup at a negligible loss while being two orders of magnitude faster than existing methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 45 references that are in the map. First posted on arXiv 4 Mar 2023.

Builds on

31 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
17 more
  1. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2020Designing Network Design Spacescited by 954
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  10. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  11. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  13. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  14. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  15. 2021Token Pooling in Vision Transformerscited by 68
  16. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  17. 2019Synchronous Transformers for End-to-End Speech Recognitioncited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2024CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inferencecited by 2
  2. 2026AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformerscited by 1
  3. 2023CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptationcited by 0
  4. 2026Later Is Better: Token Reduction for ViTs Under Distribution Shiftcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MergeOver: Post-Training Token Merging for Recursive Vision Transformers
  2. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformer
  3. 2026DORA: Dynamic Online Reinforcement Agent for Token Merging in Vision Transformers
  4. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  5. 2026Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers
  6. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  7. 2024Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
  8. 2026ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT