mlsanityData to 7 Oct 2026

TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers

arXiv 2509.03379 ↗First posted 3 Sep 2025cs.CV cs.AIcited by 0

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision Transformers (ViTs) achieve strong performance in image classification but incur high computational costs from processing all image tokens. To reduce inference costs in large ViTs without compromising accuracy, we propose TinyDrop, a training-free token dropping framework guided by a lightweight vision model. The guidance model estimates the importance of tokens while performing inference, thereby selectively discarding low-importance tokens if large vit models need to perform attention calculations. The framework operates plug-and-play, requires no architectural modifications, and is compatible with diverse ViT architectures. Evaluations on standard image classification benchmarks demonstrate that our framework reduces FLOPs by up to 80% for ViTs with minimal accuracy degradation, highlighting its generalization capability and practical utility for efficient ViT-based classification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 29 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

26 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  2. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers29 May
  3. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers27 May
  4. 2022Token Merging: Your ViT But Faster17 Oct

Other references · 22

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  6. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  7. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  8. 2022DeiT III: Revenge of the ViTcited by 417
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  11. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  12. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
10 more
  1. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  2. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  3. 2024Logit Standardization in Knowledge Distillationcited by 101
  4. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  5. 2020A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inferencecited by 61
  6. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  7. 2023Extracting Class Activation Maps from Non-Discriminative Features as wellcited by 30
  8. 2024PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inferencecited by 9
  9. 2024Tiny Models are the Computational Saver for Large Modelscited by 4
  10. 2025Optimal Brain Connection: Towards Efficient Structural Pruningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  2. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  3. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
  4. 2024Token Cropr: Faster ViTs for Quite a Few Tasks
  5. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
  6. 2026Later Is Better: Token Reduction for ViTs Under Distribution Shift
  7. 2024Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer
  8. 2025ToFe: Lagged Token Freezing and Reusing for Efficient Vision Transformer Inference