When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs
On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency
Abstract (arXiv)
Visual token pruning speeds up multimodal large language models (MLLMs) by keeping a small subset of the visual tokens, and pruning methods are compared by the accuracy they retain. We study what pruning does to the confidence of these models, across common selectors, several MLLMs, and different output formats. Pruning errors concentrate on questions whose evidence the selector removed, and confidence does not register the removal. When the queried object loses all of its tokens, accuracy on these questions drops from 59% to 17%, while confidence stays at the unpruned level. Returning a few object tokens to the kept set recovers most of the lost accuracy. Selectors that keep the most attended tokens remove such evidence most often and produce confident errors, which temperature scaling cannot re-rank. Selectors that avoid keeping redundant tokens stay close to the calibration of the unpruned model. We then use the confidence of the pruned model to set a per-question token budget. The model answers with few tokens first and again with all tokens when its confidence is low. Conformal risk control sets the threshold to bound the expected deviation from the unpruned model. With coverage-based selection, this cascade needs about a third of the prefill tokens of the unpruned model, while with FastV it needs more than the unpruned model. The savings come mainly from how well the confidence ranks the answers that differ from the unpruned ones.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 35 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
30 of 35 references.
Closest ancestors · 10
Same direction, newest first
- 2025SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs28 Oct
- 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models30 Aug
- 2025A Survey of Token Compression for Efficient Multimodal Large Language Models27 Jul
- 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs12 Jun
- 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models4 Mar
- 2025What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph4 Jan
- 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
- 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models30 Nov
- 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
- 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
Other references · 20
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023Visual Instruction Tuningcited by 7,869
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
- 2018Object Hallucination in Image Captioningcited by 669
- 2022Conformal Risk Controlcited by 299
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2024Language Model Cascades: Token-level uncertainty and beyondcited by 121
8 more
- 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
- 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
- 2024Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Modelscited by 19
- 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 12
- 2026Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volumecited by 8
- 2024Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Studycited by 5
- 2025FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMscited by 5
- 2023Uncovering the Hidden Cost of Model Compressioncited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Who Speaks for the Pruned? Visual Token Pruning as Coverage Optimization
- 2026ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs
- 2023On the Impact of Calibration Data in Post-training Quantization and Pruning
- 2026TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
- 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
- 2026IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs
- 2025GridPrune: From "Where to Look" to "What to Select" in Visual Token Pruning for MLLMs
- 2026S2Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models