# Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment

arXiv: https://arxiv.org/abs/2506.22283 · HTML: https://mlsanity.com/paper/2506.22283/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs](https://mlsanity.com/paper/2506.10967/) (2025)
- [VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models](https://mlsanity.com/paper/2505.22654/) (2025)
- [FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models](https://mlsanity.com/paper/2505.19536/) (2025)
- [How Visual Representations Map to Language Feature Space in Multimodal LLMs](https://mlsanity.com/paper/2506.11976/) (2025)

## Built on by (most cited first, 11 of 11)

- [ApET: Approximation-Error Guided Token Compression for Efficient VLMs](https://mlsanity.com/paper/2602.19870/) (2026)
- [See What Matters: Differentiable Grid Sample Pruning for Generalizable Vision-Language-Action Model](https://mlsanity.com/paper/2605.11817/) (2026)
- [Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation](https://mlsanity.com/paper/2604.12424/) (2026)
- [G2TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models](https://mlsanity.com/paper/2605.12309/) (2026)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [HiCI: Hierarchical Construction-Integration for Long-Context Attention](https://mlsanity.com/paper/2603.20843/) (2026)
- [TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models](https://mlsanity.com/paper/2511.11831/) (2025)
- [ForensicZip: More Tokens are Better but Not Necessary in Forensic Vision-Language Models](https://mlsanity.com/paper/2603.12208/) (2026)
- [Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression](https://mlsanity.com/paper/2608.02134/) (2026)
- [Conditional Visual Evidence Utility: State-Dependent Rank Reversals in Frozen Vision-Language Encoders](https://mlsanity.com/paper/2608.28316/) (2026)
- [MM-ContextFold: Context Folding for Multimodal Agentic Retrieval](https://mlsanity.com/paper/2609.23121/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) encode visual inputs as dense sequences of patch-level tokens to capture fine-grained semantics. These visual tokens often outnumber their textual counterparts by a large margin, leading to substantial computational overhead and limiting the scalability of LVLMs in practice. Previous efforts have explored visual token reduction either prior to or within the large language models (LLMs). However, most in-LLM reduction approaches rely on text-conditioned interactions, implicitly assuming that textual tokens can reliably capture the importance of visual tokens. In this work, we revisit this assumption and reveal causal, semantic, and spatial forms of cross-modal misalignment. These misalignments undermine the effectiveness of text-guided visual token reduction. To address this, we introduce VisionDrop, a training-free, visual-only pruning framework that selects informative visual tokens based on intra-modal (visual-to-visual) attention, without relying on textual signals. To further suppress redundancy throughout the model hierarchy, we treat the visual encoder and the LLM as a unified system and design a progressive pruning pipeline. Our method performs dominant token selection and lightweight contextual merging at multiple stages, enabling fine-grained visual information to be retained even under aggressive token budgets. Extensive experiments across diverse benchmarks show that VisionDrop achieves consistent improvements over existing approaches, despite requiring no additional training or complex modifications. Notably, when integrated with LLaVA-NeXT-7B, VisionDrop achieves a 2.7x reduction in inference latency and 6x in FLOPs, while retaining 95.71% of the original performance.