mlsanityData to 14 Oct 2026

When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics

arXiv 2606.03569 ↗First posted 2 Jun 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores. This single-metric paradigm presents a critical flaw: high attention scores inherently collapse onto semantically similar regions, thereby severely reducing feature diversity and discarding vital contextual details. To address this, we introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework that explicitly decouples the pruning process. The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity. The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens. This two-stage synergy constitutes the core of STS, first ensuring geometric coverage and then refining the retained tokens according to semantic relevance. Extensive evaluations demonstrate that STS mitigates the redundancy caused by attention-based selection, improving both structural diversity and fine-grained task alignment of the preserved visual tokens.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 42 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

26 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs2 Dec
  2. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
12 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2022Token Merging: Your ViT But Fastercited by 1,021
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  5. 2020Quantifying Attention Flow in Transformerscited by 775
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  7. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  8. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  9. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  10. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 52
  11. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 38
  12. 2025AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CoViST: Visual Token Compression via Composable Statescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More
  2. 2026LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models
  3. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
  4. 2026When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model
  5. 2025ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
  6. 2026AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
  7. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
  8. 2026Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models