mlsanityData to 14 Oct 2026

BlindSight: Harnessing Sparsity for Efficient Vision-Language Models

arXiv 2507.09071 ↗First posted 11 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large vision-language models (VLMs) enable joint processing of text and images. However, incorporating vision data significantly increases the prompt length, resulting in a longer time to first token (TTFT). This bottleneck can be alleviated by leveraging the inherent sparsity in the attention computation. Analyzing these attention patterns in VLMs when processing a series of images, we observe the absence of inter-image attention in a substantial portion of layers. Based on this, we propose BlindSight: an approach to optimize multi-image VLM inference using an input-template-aware attention sparsity mask with no runtime overhead. We utilize a dataset to derive a prompt-agnostic categorization for attention heads: Dense, Sink, Intra-Image, and Intra-Image+Sink. We develop a Triton-based GPU kernel to leverage this sparsity. BlindSight achieves a 1.8-3.2x speedup in the attention computation (prompt length 36K-300K). BlindSight generalizes across VLMs (Qwen2-VL, Qwen2.5-VL, Gemma 3), with only a 0.78% absolute accuracy degradation on average on multi-image comprehension benchmarks. Finally, we advocate for the design of efficient VLMs that combine BlindSight-inspired sparse and dense layers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 60 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

42 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention22 Apr
  2. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Models5 Mar
  3. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models4 Mar
  4. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More17 Feb
  5. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  6. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  7. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 35

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  12. 2025s1: Simple test-time scalingcited by 1,341
23 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  2. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  3. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  7. 2024Massive Activations in Large Language Modelscited by 294
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  9. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  10. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  11. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  12. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  13. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  14. 2025Why do LLMs attend to the first token?cited by 127
  15. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  16. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  17. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  18. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  19. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  20. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  21. 2023FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysiscited by 28
  22. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  23. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A-VL: Adaptive Attention for Large Vision-Language Models
  2. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
  3. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  4. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  5. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
  6. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  7. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
  8. 2026CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models