mlsanityData to 7 Oct 2026

Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

arXiv 2603.23914 ↗First posted 25 Mar 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 31 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

22 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference26 Jun
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
8 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  3. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  5. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  6. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  7. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  8. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  2. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies
  3. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification
  4. 2024A-VL: Adaptive Attention for Large Vision-Language Models
  5. 2026Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models
  6. 2026Large Vision-Language Models Get Lost in Attention
  7. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
  8. 2025CS-VLM: Compressed Sensing Attention for Efficient Vision-Language Representation Learning