# Making Vision Transformers Efficient from A Token Sparsification View

arXiv: https://arxiv.org/abs/2303.08685 · HTML: https://mlsanity.com/paper/2303.08685/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 30 of 52)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Co-Scale Conv-Attentional Image Transformers](https://mlsanity.com/paper/2104.06399/) (2021)

## Built on by (most cited first, 29 of 29)

- [Revisiting Token Pruning for Object Detection and Instance Segmentation](https://mlsanity.com/paper/2306.07050/) (2023)
- [HRVDA: High-Resolution Visual Document Assistant](https://mlsanity.com/paper/2404.06918/) (2024)
- [ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2406.09936/) (2024)
- [Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation](https://mlsanity.com/paper/2311.12028/) (2023)
- [Agglomerative Token Clustering](https://mlsanity.com/paper/2409.11923/) (2024)
- [AiluRus: A Scalable ViT Framework for Dense Prediction](https://mlsanity.com/paper/2311.01197/) (2023)
- [SparseDiT: Token Sparsification for Efficient Diffusion Transformer](https://mlsanity.com/paper/2412.06028/) (2024)
- [Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers](https://mlsanity.com/paper/2308.09372/) (2023)
- [CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs](https://mlsanity.com/paper/2309.15755/) (2023)
- [Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge](https://mlsanity.com/paper/2407.05941/) (2024)
- [Attention-aware Semantic Communications for Collaborative Inference](https://mlsanity.com/paper/2404.07217/) (2024)
- [Data-independent Module-aware Pruning for Hierarchical Vision Transformers](https://mlsanity.com/paper/2404.13648/) (2024)
- [H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers](https://mlsanity.com/paper/2509.06956/) (2025)
- [Morphing Tokens Draw Strong Masked Image Models](https://mlsanity.com/paper/2401.00254/) (2023)
- [Efficient Token Compression for Vision Transformer with Spatial Information Preserved](https://mlsanity.com/paper/2503.23455/) (2025)
- [Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers](https://mlsanity.com/paper/2510.21986/) (2025)
- [Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications](https://mlsanity.com/paper/2510.27186/) (2025)
- [Revisiting Vision Transformer from the View of Path Ensemble](https://mlsanity.com/paper/2308.06548/) (2023)
- [Diversity-Guided MLP Reduction for Efficient Large Vision Transformers](https://mlsanity.com/paper/2506.08591/) (2025)
- [ELIP: Efficient Discriminative Language-Image Pre-training with Fewer Vision Tokens](https://mlsanity.com/paper/2309.16738/) (2023)
- [Adaptive MLP Pruning for Large Vision Transformers](https://mlsanity.com/paper/2603.08100/) (2026)
- [ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction](https://mlsanity.com/paper/2603.26258/) (2026)
- [SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit](https://mlsanity.com/paper/2503.00060/) (2025)
- [ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT](https://mlsanity.com/paper/2602.15720/) (2026)
- [TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI](https://mlsanity.com/paper/2603.22867/) (2026)
- [ViT-FREE: Efficient Face Recognition via Early Exiting and Synthetic Adaptation](https://mlsanity.com/paper/2606.12023/) (2026)
- [BMFA: Boundary-Minority Free-Energy Adaptive Screening](https://mlsanity.com/paper/2607.17656/) (2026)
- [Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM](https://mlsanity.com/paper/2609.18279/) (2026)
- [Bottlenecked semantic compression by representation reconstruction](https://mlsanity.com/paper/2610.17412/) (2026)

## Abstract (arXiv)

The quadratic computational complexity to the number of tokens limits the practical applications of Vision Transformers (ViTs). Several works propose to prune redundant tokens to achieve efficient ViTs. However, these methods generally suffer from (i) dramatic accuracy drops, (ii) application difficulty in the local vision transformer, and (iii) non-general-purpose networks for downstream tasks. In this work, we propose a novel Semantic Token ViT (STViT), for efficient global and local vision transformers, which can also be revised to serve as backbone for downstream tasks. The semantic tokens represent cluster centers, and they are initialized by pooling image tokens in space and recovered by attention, which can adaptively represent global or local semantic information. Due to the cluster properties, a few semantic tokens can attain the same effect as vast image tokens, for both global and local vision transformers. For instance, only 16 semantic tokens on DeiT-(Tiny,Small,Base) can achieve the same accuracy with more than 100% inference speed improvement and nearly 60% FLOPs reduction; on Swin-(Tiny,Small,Base), we can employ 16 semantic tokens in each window to further speed it up by around 20% with slight accuracy increase. Besides great success in image classification, we also extend our method to video recognition. In addition, we design a STViT-R(ecover) network to restore the detailed spatial information based on the STViT, making it work for downstream tasks, which is powerless for previous token sparsification methods. Experiments demonstrate that our method can achieve competitive results compared to the original networks in object detection and instance segmentation, with over 30% FLOPs reduction for backbone. Code is available at http://github.com/changsn/STViT-R