mlsanityData to 7 Oct 2026

Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies

arXiv 2603.27960 ↗First posted 30 Mar 2026cs.LG cs.CL cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Although Large Vision Language Models (LVLMs) have demonstrated impressive multimodal reasoning capabilities, their scalability and deployment are constrained by massive computational requirements. In particular, the massive amount of visual tokens from high-resolution input data aggravates the situation due to the quadratic complexity of attention mechanisms. To address these issues, the research community has developed several optimization frameworks. This paper presents a comprehensive survey of the current state-of-the-art techniques for accelerating LVLM inference. We introduce a systematic taxonomy that categorizes existing optimization frameworks into four primary dimensions: visual token compression, memory management and serving, efficient architectural design, and advanced decoding strategies. Furthermore, we critically examine the limitations of these current methodologies and identify critical open problems to inspire future research directions in efficient multimodal systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 73 of its 80 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

73 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar

Other references · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2022Token Merging: Your ViT But Fastercited by 995
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  10. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  13. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  14. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  15. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  16. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  17. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  18. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  19. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  20. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  21. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  22. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  23. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  24. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  25. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  26. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  27. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  28. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  29. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  30. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  31. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  32. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  33. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  34. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  35. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  36. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  37. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  38. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71

50 most cited of 72 All 72 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  2. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
  3. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
  4. 2025Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
  5. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  6. 2025Towards General Continuous Memory for Vision-Language Models
  7. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
  8. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Models