mlsanityData to 7 Oct 2026

Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge

arXiv 2407.05941 ↗First posted 1 Jul 2024cs.LG cs.CVcited by 8

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

This paper investigates how to efficiently deploy vision transformers on edge devices for small workloads. Recent methods reduce the latency of transformer neural networks by removing or merging tokens, with small accuracy degradation. However, these methods are not designed with edge device deployment in mind: they do not leverage information about the latency-workload trends to improve efficiency. We address this shortcoming in our work. First, we identify factors that affect ViT latency-workload relationships. Second, we determine token pruning schedule by leveraging non-linear latency-workload relationships. Third, we demonstrate a training-free, token pruning method utilizing this schedule. We show other methods may increase latency by 2-30%, while we reduce latency by 9-26%. For similar latency (within 5.2% or 7ms) across devices we achieve 78.6%-84.5% ImageNet1K accuracy, while the state-of-the-art, Token Merging, achieves 45.8%-85.4%.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 48 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

28 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformers9 Aug
  2. 2023Learned Thresholds Token Merging and Pruning for Vision Transformers20 Jul
  3. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers27 May
  4. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers21 Apr
  5. 2022Token Merging: Your ViT But Faster17 Oct
  6. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  7. 2021Learned Token Pruning for Transformers2 Jul

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  8. 2021Intriguing Properties of Vision Transformerscited by 446
  9. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  10. 2021Post-Training Quantization for Vision Transformercited by 274
  11. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  12. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
9 more
  1. 2023Token Merging for Fast Stable Diffusioncited by 220
  2. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  3. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  4. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  5. 2020A Survey of Methods for Low-Power Deep Learning and Computer Visioncited by 37
  6. 2023Making Vision Transformers Efficient from A Token Sparsification Viewcited by 28
  7. 2022SP-ViT: Learning 2D Spatial Priors for Vision Transformerscited by 12
  8. 2023The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deploymentcited by 11
  9. 2020Towards Latency-aware DNN Optimization with GPU Runtime Analysis and Tail Effect Eliminationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attentioncited by 8
  2. 2025Recent Advances in Transformer and Large Language Models for UAV Applicationscited by 3
  3. 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Modelscited by 2
  4. 2025DINOMotion: advanced robust tissue motion tracking with DINOv2 in 2D-Cine MRI-guided radiotherapycited by 0
  5. 2026TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Modelscited by 0
  6. 2026Adversarial Decoys: Misdirecting Attention-Based Defenses in ViTcited by 0
  7. 2026Variable-Granularity Tokenization for High-Resolution Object Detectioncited by 0
  8. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EdgeFlex-Transformer: Transformer Inference for Edge Devices
  2. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  3. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
  4. 2026MergeOver: Post-Training Token Merging for Recursive Vision Transformers
  5. 2026ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
  6. 2026Dispatch-Aware Ragged Attention for Pruned Vision Transformers
  7. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  8. 2025Janus: Collaborative Vision Transformer Under Dynamic Network Environment