mlsanityData to 7 Oct 2026

R'enyi Entropy: A New Token Pruning Metric for Vision Transformers

arXiv 2603.27900 ↗First posted 29 Mar 2026cs.CVcited by 0

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Vision Transformers (ViTs) achieve state-of-the-art performance but suffer from the O(N2) complexity of self-attention, making inference costly for high-resolution inputs. To address this bottleneck, token pruning has emerged as a critical technique to accelerate inference. Most existing methods rely on the [CLS] token to estimate patch importance. However, we argue that the [CLS] token can be unreliable in early layers where semantic representations are still immature. As a result, pruning in the early layer often leads to inaccurate importance estimation and unnecessary information loss. In this work, we propose a training-free token importance metric, namely Col-Ln, which is derived from R'enyi entropy that enables the identification of informative tokens from the first layer of the network, thereby enabling more reliable pruning in token reduction. Extensive experiments on ViTs and Large Vision-Language Models (LVLMs) demonstrate that our approach consistently outperforms state-of-the-art pruning methods across diverse benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 41 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

29 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs2 Dec
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2013Describing Textures in the Wildcited by 2,110
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
15 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  11. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  12. 2022VSA: Learning Varied-Size Window Attention in Vision Transformerscited by 43
  13. 2023Multi-level Multiple Instance Learning with Transformer for Whole Slide Image Classificationcited by 6
  14. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 5
  15. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  2. 2026IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
  3. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization
  4. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  5. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  6. 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
  7. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
  8. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance