mlsanityData to 14 Oct 2026

SaiT: Sparse Vision Transformers through Adaptive Token Pruning

arXiv 2210.05832 ↗First posted 11 Oct 2022cs.CVcited by 15

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

While vision transformers have achieved impressive results, effectively and efficiently accelerating these models can further boost performances. In this work, we propose a dense/sparse training framework to obtain a unified model, enabling weight sharing across various token densities. Thus one model offers a range of accuracy and throughput tradeoffs for different applications. Besides, we introduce adaptive token pruning to optimize the patch token sparsity based on the input image. In addition, we investigate knowledge distillation to enhance token selection capability in early transformer modules. Sparse adaptive image Transformer (SaiT) offers varying levels of model acceleration by merely changing the token sparsity on the fly. Specifically, SaiT reduces the computation complexity (FLOPs) by 39% - 43% and increases the throughput by 67% - 91% with less than 0.5% accuracy loss for various vision transformer models. Meanwhile, the same model also provides the zero accuracy drop option by skipping the sparsification step. SaiT achieves better accuracy and computation tradeoffs than state-of-the-art transformer and convolutional models.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 40 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

34 of 40 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  11. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
22 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  3. 2020Designing Network Design Spacescited by 960
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  5. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  7. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  8. 2021Going deeper with Image Transformerscited by 764
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  10. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  11. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  12. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  13. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  14. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  15. 2021Post-Training Quantization for Vision Transformercited by 278
  16. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  17. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  18. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  19. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  20. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  21. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  22. 2021Centroid Transformers: Learning to Abstract with Attentioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 5; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 77
  2. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  3. 2024ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformerscited by 28
  4. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  5. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 19
  6. 2023Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimationcited by 15
  7. 2024LF-ViT: Reducing Spatial Redundancy in Vision Transformer for Efficient Image Recognitioncited by 13
  8. 2024Agglomerative Token Clusteringcited by 13
  9. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  10. 2026Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformerscited by 2
  11. 2025SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exitcited by 1
  12. 2025Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruningcited by 1
3 more
  1. 2025Two-stage Vision Transformers and Hard Masking offer Robust Object Representationscited by 1
  2. 2025ClustViT: Clustering-based Token Merging for Semantic Segmentationcited by 0
  3. 2025Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning
  3. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
  4. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
  5. 2026Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
  6. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformer
  7. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  8. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration