mlsanityData to 7 Oct 2026

MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention

arXiv 2504.16083 ↗First posted 22 Apr 2025cs.CV cs.LGcited by 26

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

The integration of long-context capabilities with visual understanding unlocks unprecedented potential for Vision Language Models (VLMs). However, the quadratic attention complexity during the pre-filling phase remains a significant obstacle to real-world deployment. To overcome this limitation, we introduce MMInference (Multimodality Million tokens Inference), a dynamic sparse attention method that accelerates the prefilling stage for long-context multi-modal inputs. First, our analysis reveals that the temporal and spatial locality of video input leads to a unique sparse pattern, the Grid pattern. Simultaneously, VLMs exhibit markedly different sparse distributions across different modalities. We introduce a permutation-based method to leverage the unique Grid pattern and handle modality boundary issues. By offline search the optimal sparse patterns for each head, MMInference constructs the sparse distribution dynamically based on the input. We also provide optimized GPU kernels for efficient sparse computations. Notably, MMInference integrates seamlessly into existing VLM pipelines without any model modifications or fine-tuning. Experiments on multi-modal benchmarks-including Video QA, Captioning, VisionNIAH, and Mixed-Modality NIAH-with state-of-the-art long-context VLMs (LongVila, LlavaVideo, VideoChat-Flash, Qwen2.5-VL) show that MMInference accelerates the pre-filling stage by up to 8.3x at 1M tokens while maintaining accuracy. Our code is available at https://aka.ms/MMInference.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 58 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

52 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  2. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration29 Oct
  3. 2024Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers17 Oct
  4. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification11 Oct
  5. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 47

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Mistral 7Bcited by 3,065
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2022Token Merging: Your ViT But Fastercited by 995
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
35 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2025Qwen2.5-Omni Technical Reportcited by 704
  3. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  5. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  6. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  7. 2024Long Context Transfer from Language to Visioncited by 502
  8. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  10. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  11. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  12. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  13. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  14. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  15. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  16. 2022Neighborhood Attention Transformercited by 235
  17. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  18. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  19. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  20. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  21. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  22. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  23. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  24. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  25. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  26. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  27. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  28. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  29. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  30. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  31. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  32. 2024DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generationcited by 26
  33. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tilecited by 26
  34. 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
  35. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 8.

26 papers.

Sort:

By area:

  1. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  2. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  3. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  4. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  5. 2025Bidirectional Sparse Attention for Faster Video Diffusion Trainingcited by 18
  6. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 13
  7. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 10
  8. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 9
  9. 2025Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generationcited by 5
  10. 2025TimeScope: Towards Task-Oriented Temporal Grounding In Long Videoscited by 5
  11. 2025Sparser Block-Sparse Attention via Token Permutationcited by 5
  12. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
14 more
  1. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understandingcited by 4
  2. 2025Sparsity Forcing: Reinforcing Token Sparsity of MLLMscited by 3
  3. 2026POP: Prefill-Only Pruning for Efficient Large Model Inferencecited by 2
  4. 2026VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inferencecited by 2
  5. 2025AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference with Dynamical Text Guidancecited by 1
  6. 2025MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Trainingcited by 1
  7. 2025OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMscited by 1
  8. 2026S2O: Early Stopping for Sparse Attention via Online Permutationcited by 1
  9. 2025BlindSight: Harnessing Sparsity for Efficient Vision-Language Modelscited by 0
  10. 2025The Hybrid Multimodal Graph Index (HMGI): A Comprehensive Framework for Integrated Relational and Vector Searchcited by 0
  11. 2025Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directionscited by 0
  12. 2026APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attentioncited by 0
  13. 2026AdaSpark: Adaptive Sparsity for Efficient Long-Video Understandingcited by 0
  14. 2026HHR: Hierarchical Hash Retrieval for Efficient LLM Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
  2. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
  3. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
  4. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
  5. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
  6. 2025MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models
  7. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
  8. 2025CS-VLM: Compressed Sensing Attention for Efficient Vision-Language Representation Learning