mlsanityData to 14 Oct 2026

SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference

arXiv 2510.17777 ↗First posted 20 Oct 2025cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision Language Models (VLMs) have rapidly advanced in integrating visual and textual reasoning, powering applications across high-resolution image understanding, long-video analysis, and multi-turn conversation. However, their scalability remains limited by the growing number of visual tokens that dominate inference latency. We present SparseVILA, a new paradigm for efficient VLM inference that decouples visual sparsity across the prefilling and decoding stages. SparseVILA distributes sparsity across stages by pruning redundant visual tokens during prefill and retrieving only query-relevant tokens during decoding. This decoupled design matches leading prefill pruning methods while preserving multi-turn fidelity by retaining most of the visual cache so that query-aware tokens can be retrieved at each conversation round. Built on an AWQ-optimized inference pipeline, SparseVILA achieves up to 4.0 times faster prefilling, 2.5 times faster decoding, and an overall 2.6 times end-to-end speedup on long-context video tasks -- while improving accuracy on document-understanding and reasoning tasks. By decoupling query-agnostic pruning and query-aware retrieval, SparseVILA establishes a new direction for efficient multimodal inference, offering a training-free, architecture-agnostic framework for accelerating large VLMs without sacrificing capability.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 76 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

64 of 76 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Models5 Mar
  2. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?17 Feb
  3. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  4. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  5. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  6. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  7. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 56

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2019Towards VQA Models That Can Readcited by 1,718
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  8. 2022Token Merging: Your ViT But Fastercited by 1,021
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2016A Diagram Is Worth A Dozen Imagescited by 804
  13. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  14. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  15. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  16. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  17. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  18. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  19. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  20. 2024Long Context Transfer from Language to Visioncited by 515
  21. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  22. 2021InfographicVQAcited by 437
  23. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  24. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  25. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  26. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  27. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  28. 2023Token Merging for Fast Stable Diffusioncited by 224
  29. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  30. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  31. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  32. 2021Learned Token Pruning for Transformerscited by 139
  33. 2021Patch Slimming for Efficient Vision Transformerscited by 133
  34. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
  35. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  36. 2025Scaling RL to Long Videoscited by 84
  37. 2023Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformerscited by 81
  38. 2021Vision Transformer Pruningcited by 79

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 14.

15 papers.

Sort:

By area:

  1. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  2. 2026Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decodingcited by 7
  3. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  4. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  5. 2026Stateful Token Reduction for Long-Video Hybrid VLMscited by 4
  6. 2026Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Modelscited by 4
  7. 2026Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMscited by 3
  8. 2026Visual Access Boundaries in Vision-Language Model Reasoningcited by 2
  9. 2026MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Modelscited by 1
  10. 2026Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turnscited by 1
  11. 2026VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-Language Modelscited by 0
  12. 2026Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inferencecited by 0
3 more
  1. 2026CIVIC: End-to-End Sequence Compactness for Efficient Vision-Language Modelscited by 0
  2. 2026MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Modelscited by 0
  3. 2026TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  3. 2025Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference
  4. 2026OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning
  5. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
  6. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  7. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass
  8. 2026Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression