# Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers

arXiv: https://arxiv.org/abs/2305.17328 · HTML: https://mlsanity.com/paper/2305.17328/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving.
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 27 May 2023.

## Placed in

- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [All Tokens Matter: Token Labeling for Training Better Vision Transformers](https://mlsanity.com/paper/2104.10858/) (2021)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [Differentiable Ranks and Sorting using Optimal Transport](https://mlsanity.com/paper/1905.11885/) (2019)
- [Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers](https://mlsanity.com/paper/2304.10716/) (2023)

## Built on by (most cited first, 30 of 62)

- [Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models](https://mlsanity.com/paper/2409.10197/) (2024)
- [AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning](https://mlsanity.com/paper/2412.03248/) (2024)
- [Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models](https://mlsanity.com/paper/2405.05252/) (2024)
- [Accelerating Transformers with Spectrum-Preserving Token Merging](https://mlsanity.com/paper/2405.16148/) (2024)
- [Similarity-Aware Token Pruning: Your VLM but Faster](https://mlsanity.com/paper/2503.11549/) (2025)
- [ALGM: Adaptive Local-then-Global Token Merging for Efficient Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2406.09936/) (2024)
- [Importance-Based Token Merging for Efficient Image and Video Generation](https://mlsanity.com/paper/2411.16720/) (2024)
- [The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating](https://mlsanity.com/paper/2603.07135/) (2026)
- [PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference](https://mlsanity.com/paper/2407.05010/) (2024)
- [Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge](https://mlsanity.com/paper/2407.05941/) (2024)
- [FOCUS: Fused Observation of Channels for Unveiling Spectra](https://mlsanity.com/paper/2507.14787/) (2025)
- [TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning](https://mlsanity.com/paper/2507.22872/) (2025)
- [VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference](https://mlsanity.com/paper/2508.17857/) (2025)
- [Representation Shift: Unifying Token Compression with FlashAttention](https://mlsanity.com/paper/2508.00367/) (2025)
- [CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling](https://mlsanity.com/paper/2406.12018/) (2024)
- [Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer](https://mlsanity.com/paper/2408.17062/) (2024)
- [Is Less More? Exploring Token Condensation as Training-free Test-time Adaptation](https://mlsanity.com/paper/2410.14729/) (2024)
- [Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable](https://mlsanity.com/paper/2412.02054/) (2024)
- [Efficient Token Compression for Vision Transformer with Spatial Information Preserved](https://mlsanity.com/paper/2503.23455/) (2025)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)
- [How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?](https://mlsanity.com/paper/2511.05449/) (2025)
- [Enhancing Layer Attention Efficiency through Pruning Redundant Retrievals](https://mlsanity.com/paper/2503.06473/) (2025)
- [Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models](https://mlsanity.com/paper/2503.08019/) (2025)
- [Lossless Token Merging Even Without Fine-Tuning in Vision Transformers](https://mlsanity.com/paper/2505.15160/) (2025)
- [Diversity-Guided MLP Reduction for Efficient Large Vision Transformers](https://mlsanity.com/paper/2506.08591/) (2025)
- [Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruning](https://mlsanity.com/paper/2605.20551/) (2026)
- [SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models](https://mlsanity.com/paper/2609.01004/) (2026)
- [Accelerate 3D Object Detection Models via Zero-Shot Attention Key Pruning](https://mlsanity.com/paper/2503.08101/) (2025)
- [Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding](https://mlsanity.com/paper/2507.09334/) (2025)
- [Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models](https://mlsanity.com/paper/2512.01949/) (2025)

## Abstract (arXiv)

Deployment of Transformer models on edge devices is becoming increasingly challenging due to the exponentially growing inference cost that scales quadratically with the number of tokens in the input sequence. Token pruning is an emerging solution to address this challenge due to its ease of deployment on various Transformer backbones. However, most token pruning methods require computationally expensive fine-tuning, which is undesirable in many edge deployment cases. In this work, we propose Zero-TPrune, the first zero-shot method that considers both the importance and similarity of tokens in performing token pruning. It leverages the attention graph of pre-trained Transformer models to produce an importance distribution for tokens via our proposed Weighted Page Rank (WPR) algorithm. This distribution further guides token partitioning for efficient similarity-based pruning. Due to the elimination of the fine-tuning overhead, Zero-TPrune can prune large models at negligible computational cost, switch between different pruning configurations at no computational cost, and perform hyperparameter tuning efficiently. We evaluate the performance of Zero-TPrune on vision tasks by applying it to various vision Transformer backbones and testing them on ImageNet. Without any fine-tuning, Zero-TPrune reduces the FLOPs cost of DeiT-S by 34.7% and improves its throughput by 45.3% with only 0.4% accuracy loss. Compared with state-of-the-art pruning methods that require fine-tuning, Zero-TPrune not only eliminates the need for fine-tuning after pruning but also does so with only 0.1% accuracy loss. Compared with state-of-the-art fine-tuning-free pruning methods, Zero-TPrune reduces accuracy loss by up to 49% with similar FLOPs budgets. Project webpage: https://jha-lab.github.io/zerotprune.