# Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge

arXiv: https://arxiv.org/abs/2407.05941 · HTML: https://mlsanity.com/paper/2407.05941/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving.
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Scatterbrain: Unifying Sparse and Low-rank Attention Approximation](https://mlsanity.com/paper/2110.15343/) (2021)
- [IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers](https://mlsanity.com/paper/2106.12620/) (2021)
- [Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers](https://mlsanity.com/paper/2304.10716/) (2023)
- [Learning to Merge Tokens in Vision Transformers](https://mlsanity.com/paper/2202.12015/) (2022)
- [Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers](https://mlsanity.com/paper/2305.17328/) (2023)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)
- [A Survey of Methods for Low-Power Deep Learning and Computer Vision](https://mlsanity.com/paper/2003.11066/) (2020)
- [Making Vision Transformers Efficient from A Token Sparsification View](https://mlsanity.com/paper/2303.08685/) (2023)
- [Learned Thresholds Token Merging and Pruning for Vision Transformers](https://mlsanity.com/paper/2307.10780/) (2023)
- [SP-ViT: Learning 2D Spatial Priors for Vision Transformers](https://mlsanity.com/paper/2206.07662/) (2022)
- [The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment](https://mlsanity.com/paper/2302.06117/) (2023)
- [Towards Latency-aware DNN Optimization with GPU Runtime Analysis and Tail Effect Elimination](https://mlsanity.com/paper/2011.03897/) (2020)

## Built on by (most cited first, 9 of 9)

- [ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention](https://mlsanity.com/paper/2602.07574/) (2026)
- [Recent Advances in Transformer and Large Language Models for UAV Applications](https://mlsanity.com/paper/2508.11834/) (2025)
- [StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models](https://mlsanity.com/paper/2603.07307/) (2026)
- [DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapy](https://mlsanity.com/paper/2508.10260/) (2025)
- [TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models](https://mlsanity.com/paper/2607.04593/) (2026)
- [Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT](https://mlsanity.com/paper/2607.07922/) (2026)
- [Variable-Granularity Tokenization for High-Resolution Object Detection](https://mlsanity.com/paper/2608.28706/) (2026)
- [Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TM](https://mlsanity.com/paper/2609.18279/) (2026)
- [Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2610.18858/) (2026)

## Abstract (arXiv)

This paper investigates how to efficiently deploy vision transformers on edge devices for small workloads. Recent methods reduce the latency of transformer neural networks by removing or merging tokens, with small accuracy degradation. However, these methods are not designed with edge device deployment in mind: they do not leverage information about the latency-workload trends to improve efficiency. We address this shortcoming in our work. First, we identify factors that affect ViT latency-workload relationships. Second, we determine token pruning schedule by leveraging non-linear latency-workload relationships. Third, we demonstrate a training-free, token pruning method utilizing this schedule. We show other methods may increase latency by 2-30%, while we reduce latency by 9-26%. For similar latency (within 5.2% or 7ms) across devices we achieve 78.6%-84.5% ImageNet1K accuracy, while the state-of-the-art, Token Merging, achieves 45.8%-85.4%.