mlsanityData to 7 Oct 2026

Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer

arXiv 2408.17062 ↗First posted 30 Aug 2024cs.CVcited by 4

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Despite the remarkable success of Vision Transformers (ViTs) in various visual tasks, they are often hindered by substantial computational cost. In this work, we introduce Vote&Mix (VoMix), a plug-and-play and parameter-free token reduction method, which can be readily applied to off-the-shelf ViT models textitwithout any training. VoMix tackles the computational redundancy of ViTs by identifying tokens with high homogeneity through a layer-wise token similarity voting mechanism. Subsequently, the selected tokens are mixed into the retained set, thereby preserving visual information. Experiments demonstrate VoMix significantly improves the speed-accuracy tradeoff of ViTs on both images and videos. Without any training, VoMix achieves a 2× increase in throughput of existing ViT-H on ImageNet-1K and a 2.4× increase in throughput of existing ViT-L on Kinetics-400 video dataset, with a mere 0.3% drop in top-1 accuracy.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 46 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

43 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging2 Dec
  2. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformers9 Aug
  3. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers27 May
  4. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers21 Apr
  5. 2022Token Merging: Your ViT But Faster17 Oct
  6. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2021Video Swin Transformercited by 1,090
25 more
  1. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  2. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  3. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  5. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  6. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  7. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  8. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  9. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  10. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  11. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  12. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  13. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  14. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  15. 2022Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformercited by 99
  16. 2023What Do Self-Supervised Vision Transformers Learn?cited by 90
  17. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  18. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  19. 2022Learning to Merge Tokens in Vision Transformerscited by 64
  20. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  21. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformerscited by 47
  22. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  23. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  24. 2021Self-slimmed Vision Transformercited by 27
  25. 2023Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Modelscited by 3
  2. 2025Training-free Token Reduction for Vision Mambacited by 1
  3. 2025Towards Efficient Vision State Space Models via Token Mergingcited by 0
  4. 2026CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  2. 2026Later Is Better: Token Reduction for ViTs Under Distribution Shift
  3. 2026MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis
  4. 2025TinyDrop: Tiny Model Guided Token Dropping for Vision Transformers
  5. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
  6. 2023No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
  7. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
  8. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformers