mlsanityData to 7 Oct 2026

SPViT: Enabling Faster Vision Transformers via Soft Token Pruning

arXiv 2112.13890 ↗First posted 27 Dec 2021cs.CV cs.AI cs.ARcited by 165

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Recently, Vision Transformer (ViT) has continuously established new milestones in the computer vision field, while the high computation and memory cost makes its propagation in industrial production difficult. Pruning, a traditional model compression paradigm for hardware efficiency, has been widely applied in various DNN structures. Nevertheless, it stays ambiguous on how to perform exclusive pruning on the ViT structure. Considering three key points: the structural characteristics, the internal data pattern of ViTs, and the related edge device deployment, we leverage the input token sparsity and propose a computation-aware soft pruning framework, which can be set up on vanilla Transformers of both flatten and CNN-type structures, such as Pooling-based ViT (PiT). More concretely, we design a dynamic attention-based multi-head token selector, which is a lightweight module for adaptive instance-wise token selection. We further introduce a soft pruning technique, which integrates the less informative tokens generated by the selector module into a package token that will participate in subsequent calculations rather than being completely discarded. Our framework is bound to the trade-off between accuracy and computation constraints of specific edge devices through our proposed computation-aware training strategy. Experimental results show that our framework significantly reduces the computation cost of ViTs while maintaining comparable performance on image classification. Moreover, our framework can guarantee the identified model to meet resource specifications of mobile devices and FPGA, and even achieve the real-time execution of DeiT-T on mobile platforms. For example, our method reduces the latency of DeiT-T to 26 ms (26%~ 41% superior to existing works) on the mobile device with 0.25%~ 4% higher top-1 accuracy on ImageNet.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 91 of its 111 references that are in the map. First posted on arXiv 27 Dec 2021.

Builds on

91 of 111 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 90

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2017Squeeze-and-Excitation Networkscited by 4,763
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  10. 2020Point Transformercited by 1,462
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
38 more
  1. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  2. 2021Scaling Vision Transformerscited by 993
  3. 2020Designing Network Design Spacescited by 954
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  5. 2020Pre-Trained Image Processing Transformercited by 891
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  8. 2020PCT: Point cloud transformercited by 730
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  10. 2021Transformer in Transformercited by 632
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  12. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  13. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  14. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  15. 2021Transformer Trackingcited by 412
  16. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  17. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  18. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  19. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  20. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  21. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  22. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  23. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  24. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  25. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  26. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  27. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  28. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  29. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  30. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  31. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  32. 2020Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformerscited by 190
  33. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  34. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  35. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  36. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  37. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  38. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 23; 2024: 60; 2025: 44; 2026: 30.

165 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  4. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  6. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 79
  7. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  8. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  9. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  10. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  11. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  12. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
38 more
  1. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  2. 2022Interaction Transformer for Human Reaction Generationcited by 42
  3. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 42
  4. 2022TORE: Token Reduction for Efficient Human Mesh Recovery with Transformercited by 41
  5. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  6. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  7. 2024Scene Adaptive Sparse Transformer for Event-based Object Detectioncited by 39
  8. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 39
  9. 2024MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Maskscited by 39
  10. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  11. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  12. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 35
  13. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  14. 2022Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuningcited by 33
  15. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  16. 2024Token Caching for Diffusion Transformer Accelerationcited by 29
  17. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  18. 2024Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformerscited by 26
  19. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  20. 2023SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformerscited by 24
  21. 2024Numerical Pruning for Efficient Autoregressive Modelscited by 24
  22. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  23. 2023Skip-Attention: Improving Vision Transformers by Paying Less Attentioncited by 23
  24. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  25. 2023How can objects help action recognition?cited by 21
  26. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  27. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19
  28. 2023GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagationcited by 19
  29. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  30. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  31. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  32. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  33. 2022Dynamic Focus-aware Positional Queries for Semantic Segmentationcited by 16
  34. 2024Squat: Quant Small Language Models on the Edgecited by 16
  35. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  36. 2024LPViT: Low-Power Semi-structured Pruning for Vision Transformerscited by 15
  37. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  38. 2024Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessmentcited by 13

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  2. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
  3. 2022SaiT: Sparse Vision Transformers through Adaptive Token Pruning
  4. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  5. 2024Token Pruning using a Lightweight Background Aware Vision Transformer
  6. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers
  7. 2026Dispatch-Aware Ragged Attention for Pruned Vision Transformers
  8. 2024Exploring Token Pruning in Vision State Space Models